Re: [Wikitech-l] ResourceLoader Debug Mode
Tim, First off, thank you for taking the time to respond here, I know you have a lot on your plate right now, and I hope things are going well for you. I did not mean to call you out as having made a mistake as much as I wanted to point to examples of what kinds of changes have been made based on what I feel are misunderstandings. On 9/29/10 6:28 PM, Tim Starling wrote: On 30/09/10 08:27, Trevor Parscal wrote: There seems to be some confusion about how ResourceLoader works, which has been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification. I made that change because it was requested by multiple people in discussions before the resource loader was implemented. It's not because I actually had any actual problem with debugging, I'm well aware of the existence of the debug mode. Your responsiveness to the community is admirable, but I feel this may be a case where more collaborative discussion should have taken place before action was taken. Nonetheless, I appreciate your boldness, even if I disagree with the action taken. Concerns were raised that it may be necessary to interpret minified code in cases such as: * Where it is suspected that, due to caching issues, the debug code may not be the same as the minified code. Shift+refresh? * To debug the minifier itself, which is far short of a full JavaScript parser and may well introduce functional differences. If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks. * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge. We should probably calculate just how small that cost is before we start making asumptions based on it being fairly small. Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers. I am not meaning to argue with *you* on a point-by-point basis as much as I'm arguing with the originators of these points. I get the feeling that, while you may understand how resource loader works, they may not. My top priorities are to make the site as high-performance as possible while maintaining a reasonable level of developer friendliness. These are on opposite ends of the spectrum, and middle ground serves neither well. - Trevor -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
2010/9/30 Trevor Parscal tpars...@wikimedia.org: * Where it is suspected that, due to caching issues, the debug code may not be the same as the minified code. Shift+refresh? Or change the URL in some way that doesn't alter the meaning of the URL. There's various ways to get around caching. * To debug the minifier itself, which is far short of a full JavaScript parser and may well introduce functional differences. If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks. JSMin causing a bug sounds very unlikely to me, but I agree that we should be paranoid and account for the possibility anyway. Still, I think we can agree that JSMin breaking something should be a very, very rare occurrence (it's been around for a long time and has been used on a large body of code), and that debugging the minifier would be done locally anyway (to be sure caching is not to blame), so line breaks in production won't help. All you'd need to debug JSMin locally without going insane is to be able to preserve line breaks locally, which can be accommodated with a setting (off by default) or even just requiring commenting out of code (as long as it's clear how to do this and not cumbersome, this should be fine). but depending on the user's preferences they may be totally different from one user to another. This is false in almost all cases. Because we make our requests as cacheable as possible, the vast majority of RL responses will *not* depend on user preferences, or in fact on anything not in the URL. For these requests, you'll be able to pull up the same URL that the user is pulling and see the exact same thing (provided you both clear your caches). User preferences and other factors may be used to *build* the URL, so whether you see the issue may depend on your preferences, but once you find out which URL the user is getting their stuff from you'll have a meaningful line number. There should only be one request whose contents depend on user preferences: the one getting the user's preferences plus the user scripts. The former is really just one call to a setter with a JSON-encoded preferences map, so it consists of one simple statement; inspecting that is doable. The rest is the user's own user JS, which can contain almost anything. This is the only thing you won't reasonably be able to debug based on just a line number, but then there's lots of other easy ways to nail down user JS as being the culprit. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. Per the above this is not entirely true, but the one case in which they are useful (culprit is in user JS+prefs request and all you have is a line number) doesn't warrant preserving line breaks IMO. We should probably calculate just how small that cost is before we start making asumptions based on it being fairly small. I agree that we should get our numbers first, but some preliminary playing around seems to indicate Tim's right about this one. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
On 9/30/10 10:11 AM, Roan Kattouw wrote: 2010/9/30 Trevor Parscaltpars...@wikimedia.org: * Where it is suspected that, due to caching issues, the debug code may not be the same as the minified code. Shift+refresh? Or change the URL in some way that doesn't alter the meaning of the URL. There's various ways to get around caching. * To debug the minifier itself, which is far short of a full JavaScript parser and may well introduce functional differences. If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks. JSMin causing a bug sounds very unlikely to me, but I agree that we should be paranoid and account for the possibility anyway. Still, I think we can agree that JSMin breaking something should be a very, very rare occurrence (it's been around for a long time and has been used on a large body of code), and that debugging the minifier would be done locally anyway (to be sure caching is not to blame), so line breaks in production won't help. All you'd need to debug JSMin locally without going insane is to be able to preserve line breaks locally, which can be accommodated with a setting (off by default) or even just requiring commenting out of code (as long as it's clear how to do this and not cumbersome, this should be fine). but depending on the user's preferences they may be totally different from one user to another. This is false in almost all cases. Because we make our requests as cacheable as possible, the vast majority of RL responses will *not* depend on user preferences, or in fact on anything not in the URL. For these requests, you'll be able to pull up the same URL that the user is pulling and see the exact same thing (provided you both clear your caches). User preferences and other factors may be used to *build* the URL, so whether you see the issue may depend on your preferences, but once you find out which URL the user is getting their stuff from you'll have a meaningful line number. There should only be one request whose contents depend on user preferences: the one getting the user's preferences plus the user scripts. The former is really just one call to a setter with a JSON-encoded preferences map, so it consists of one simple statement; inspecting that is doable. The rest is the user's own user JS, which can contain almost anything. This is the only thing you won't reasonably be able to debug based on just a line number, but then there's lots of other easy ways to nail down user JS as being the culprit. Yes, thank you for correcting me here. In fact, we encode the user's name into the URL (but still limit access to the information to those with a matching session) so the URLs are all always unique to the response. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. Per the above this is not entirely true, but the one case in which they are useful (culprit is in user JS+prefs request and all you have is a line number) doesn't warrant preserving line breaks IMO. We should probably calculate just how small that cost is before we start making asumptions based on it being fairly small. I agree that we should get our numbers first, but some preliminary playing around seems to indicate Tim's right about this one. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
On 9/30/10 9:31 AM, Trevor Parscal wrote: Tim, First off, thank you for taking the time to respond here, I know you have a lot on your plate right now, and I hope things are going well for you. I did not mean to call you out as having made a mistake as much as I wanted to point to examples of what kinds of changes have been made based on what I feel are misunderstandings. On 9/29/10 6:28 PM, Tim Starling wrote: On 30/09/10 08:27, Trevor Parscal wrote: There seems to be some confusion about how ResourceLoader works, which has been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification. I made that change because it was requested by multiple people in discussions before the resource loader was implemented. It's not because I actually had any actual problem with debugging, I'm well aware of the existence of the debug mode. Your responsiveness to the community is admirable, but I feel this may be a case where more collaborative discussion should have taken place before action was taken. Nonetheless, I appreciate your boldness, even if I disagree with the action taken. Concerns were raised that it may be necessary to interpret minified code in cases such as: * Where it is suspected that, due to caching issues, the debug code may not be the same as the minified code. Shift+refresh? * To debug the minifier itself, which is far short of a full JavaScript parser and may well introduce functional differences. If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks. * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge. We should probably calculate just how small that cost is before we start making asumptions based on it being fairly small. Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers. OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. I don't know what that kind of bandwidth costs the foundation, but it's not free. I am not meaning to argue with *you* on a point-by-point basis as much as I'm arguing with the originators of these points. I get the feeling that, while you may understand how resource loader works, they may not. My top priorities are to make the site as high-performance as possible while maintaining a reasonable level of developer friendliness. These are on opposite ends of the spectrum, and middle ground serves neither well. - Trevor -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] ResourceLoader timestamp format
Early on in the requirements stage of ResourceLoader development we decided to use ISO8601 as the format for representing timestamps in URLs. This was chosen for it's legibility, conformance to a standard and ease of generation. However this was somewhat of an oversight since the timestamp 1970-01-01T00:00:00Z gets URL encoded to be 1970-01-01T00%3A00%3A00Z which leaves something to be desired. Also, generating this format in JavaScript requires sending a extra 220 bytes (minified and compressed). So, before we seal the deal on using 8601, I would like to collect some ideas about alternatives which would ideally... * Be legible in a URL * Conform to a well-defined/well-known standard * Be easy to generate from a unix timestamp in both PHP and JavaScript Proposals wanted. - Trevor ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader timestamp format
ISO 8601 does not mandate the use of separators within date and time. The following are equivalent. basic format: 19850412T101530Z extended format: 1985-04-12T10:15:30Z On 9/30/10 12:30 PM, Trevor Parscal wrote: Early on in the requirements stage of ResourceLoader development we decided to use ISO8601 as the format for representing timestamps in URLs. This was chosen for it's legibility, conformance to a standard and ease of generation. However this was somewhat of an oversight since the timestamp 1970-01-01T00:00:00Z gets URL encoded to be 1970-01-01T00%3A00%3A00Z which leaves something to be desired. Also, generating this format in JavaScript requires sending a extra 220 bytes (minified and compressed). So, before we seal the deal on using 8601, I would like to collect some ideas about alternatives which would ideally... * Be legible in a URL * Conform to a well-defined/well-known standard * Be easy to generate from a unix timestamp in both PHP and JavaScript Proposals wanted. - Trevor ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l -- Neil Kandalgaonkar (| ne...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader timestamp format
Neil Kandalgaonkar wrote: ISO 8601 does not mandate the use of separators within date and time. The following are equivalent. basic format: 19850412T101530Z extended format: 1985-04-12T10:15:30Z I'm not sure I'd call the basic format very readable. Internally, MediaWiki largely uses MMDDhhmmss stored as a string, not as a timestamp. I think this is fairly readable personally, but it's obviously breaking standards. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader timestamp format
Well, the MediaWiki timestamp string (19850412101530) is very similar to the basic ISO8601 format (19850412T101530Z), the difference being the use of T as a separator and the Z at the end to indicate GMT. I guess I'm a little confused why someone would find 19850412101530 to be fairly readable while also finding 19850412T101530Z to be not very readable. - Trevor On 9/30/10 12:56 PM, MZMcBride wrote: Neil Kandalgaonkar wrote: ISO 8601 does not mandate the use of separators within date and time. The following are equivalent. basic format: 19850412T101530Z extended format: 1985-04-12T10:15:30Z I'm not sure I'd call the basic format very readable. Internally, MediaWiki largely uses MMDDhhmmss stored as a string, not as a timestamp. I think this is fairly readable personally, but it's obviously breaking standards. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
At 2010-09-30 20:35, Trevor Parscal wrote: [...] * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this: 1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code... The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch. Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge. We should probably calculate just how small that cost is before we start making asumptions based on it being fairly small. Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers. OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. I don't know what that kind of bandwidth costs the foundation, but it's not free. OK. It's not free, the question is it worth the effort and are you really calculating this correctly? I think it would be worth it as a just-in-case thing. Also JS is usually cached and actually get stuck in cache for days. And just to be sure - did you calculated gziped version difference or plain? Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too). Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-). Regards, Nux. ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader timestamp format
2010/9/30 Neil Kandalgaonkar ne...@wikimedia.org: ISO 8601 does not mandate the use of separators within date and time. The following are equivalent. basic format: 19850412T101530Z extended format: 1985-04-12T10:15:30Z Does it also allow replacing the separators with whatever you want? Colons get urlencoded but dashes are fine, so what about (and I know I'm stealing this suggestion from /someone/, I just can't remember who suggested it to me) 1985-04-12T10-15-30Z ? Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
On 9/30/10 1:55 PM, Maciej Jaros wrote: At 2010-09-30 20:35, Trevor Parscal wrote: [...] * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this: 1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code... The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch. There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than error on line ##. Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge. We should probably calculate just how small that cost is before we start making asumptions based on it being fairly small. Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers. OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. I don't know what that kind of bandwidth costs the foundation, but it's not free. OK. It's not free, the question is it worth the effort and are you really calculating this correctly? I think it would be worth it as a just-in-case thing. Also JS is usually cached and actually get stuck in cache for days. And just to be sure - did you calculated gziped version difference or plain? Compressed of course. It would be 5-10 times that much without compression. Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too). This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code. Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-). It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code. I agree that something will always break - I just disagree that there's much merit in your approach to solving this. If we are talking about allowing the ResourceLoader to be configurable with LocalSettings.php adjustments, then you are already assuming a level of expertise about the developer debugging the problem that these middle-ground approaches will be useless. If we are talking about adding yet another URL parameter to be able to turn
Re: [Wikitech-l] ResourceLoader timestamp format
As of r74042 a new format is supported by wfTimestamp called TS_ISO8601_BASIC. This is being used by ResourceLoader now. Thank you to everyone for contributing your ideas. - Trevor On 9/30/10 2:12 PM, Roan Kattouw wrote: 2010/9/30 Neil Kandalgaonkarne...@wikimedia.org: ISO 8601 does not mandate the use of separators within date and time. The following are equivalent. basic format: 19850412T101530Z extended format: 1985-04-12T10:15:30Z Does it also allow replacing the separators with whatever you want? Colons get urlencoded but dashes are fine, so what about (and I know I'm stealing this suggestion from /someone/, I just can't remember who suggested it to me) 1985-04-12T10-15-30Z ? Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
2010/9/30 Maciej Jaros e...@wp.pl: The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. Minified code does not contain zero line breaks. Some line breaks are left in, although I don't understand exactly what the logic is there. Looking at a large minified file, the longest line I see is 1700ish characters, and that one contains a long array of strings; 'real' code lines don't seem to exceed 1000 characters, and most seem to be below 600. It's true that debuggers struggle with this. Firebug has no problem with jumping from one statement to the next if there's a dozen statements on one line, but of course setting breakpoints is still line-based so that's not very helpful. However, Trevor's main point is that you shouldn't be /trying/ to read or debug minified code, because we have a debug mode to give you unminified (and uncombined) code. Roan Kattouw (Catrope) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader timestamp format
On 9/30/10 2:12 PM, Roan Kattouw wrote: Does it also allow replacing the separators with whatever you want? If you use a separator for time, ISO 8601:2004 section 3.4.4 says it has to be hyphen for year, month, day, and colon for hour, minute, and second. -- Neil Kandalgaonkar (| ne...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
At 2010-09-30 23:13, Trevor Parscal wrote: On 9/30/10 1:55 PM, Maciej Jaros wrote: At 2010-09-30 20:35, Trevor Parscal wrote: [...] * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this: 1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code... The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch. There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than error on line ##. It's not an assumption. I AM one of the main developers working on bugs reported by users on pl.wiki. And yes line number is very helpful if accurate. Also note that I was picturing a situation where I have to debug the minified code when I don't get the error in my browser(s) at hand. Usually the problem is some extension/plug-in, but I can't tell unless I investigate. And reading minified code at it's current state is impossible for me. I understand the need to minify code I really do. I just don't get it why do you want to minify this that much. HTML which is usually served at every page load is not minified. Further more even minified HTML can be readable for debuging purposes as every debugger (inspector) shows the structure of the code rather then a code pulp. Also why are there are explicitly added comments like !-- /navigation --. People don't need comments and whitespace to read code, right? Also I have a PHP minifier if you are interested. Let's serve Mediawiki encode and if some developer want to debug the code he can always switch folders. Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too). This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code. Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-). It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code. This is impossible. Fully optimized code would only contain one byte per variable name. Yo need to make sacrifices. And this seems to be needed as shown before and above. I agree that something will always break - I just disagree that there's much merit in your approach to solving this. If we are talking about allowing the ResourceLoader to be configurable with LocalSettings.php adjustments, then you are already assuming a level of expertise about the developer debugging the problem that these middle-ground approaches will be useless. If we are talking about adding yet another URL parameter to be able to turn minification off, or otherwise customize the processing of the modules beyond turning debug on and off, I think you are adding complexity without reasonable benefit. Well we differentiate
Re: [Wikitech-l] ResourceLoader Debug Mode
On 9/30/10 4:31 PM, Maciej Jaros wrote: At 2010-09-30 23:13, Trevor Parscal wrote: On 9/30/10 1:55 PM, Maciej Jaros wrote: At 2010-09-30 20:35, Trevor Parscal wrote: [...] * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this: 1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code... The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch. There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than error on line ##. It's not an assumption. I AM one of the main developers working on bugs reported by users on pl.wiki. And yes line number is very helpful if accurate. Also note that I was picturing a situation where I have to debug the minified code when I don't get the error in my browser(s) at hand. Usually the problem is some extension/plug-in, but I can't tell unless I investigate. And reading minified code at it's current state is impossible for me. How can you possibly resolve an issue if you can't reproduce it? Developers have access to multiple browsers for this exact reason. Otherwise, you are debugging and programming blind, which is extremely unproductive. I understand the need to minify code I really do. I just don't get it why do you want to minify this that much. HTML which is usually served at every page load is not minified. Further more even minified HTML can be readable for debuging purposes as every debugger (inspector) shows the structure of the code rather then a code pulp. Also why are there are explicitly added comments like !-- /navigation --. People don't need comments and whitespace to read code, right? Also I have a PHP minifier if you are interested. Let's serve Mediawiki encode and if some developer want to debug the code he can always switch folders. I think that our outgoing HTML *should* be minified, and those comments should be stripped, I just have yet to get to that as of yet. Go to www.google.com and view source. Tell me what you see. Why are we so different from them? They are taking advantage of aggressive minification, and we should too. To be honest, I can't quite wrap my head around why I am getting any friction on making the front-end as fast as possible. I was expecting people to say I had not gone far enough. Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too). This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code. Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-). It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code. This is impossible. Fully optimized code would only
Re: [Wikitech-l] ResourceLoader Debug Mode
At 2010-10-01 01:50, Trevor Parscal wrote: On 9/30/10 4:31 PM, Maciej Jaros wrote: At 2010-09-30 23:13, Trevor Parscal wrote: On 9/30/10 1:55 PM, Maciej Jaros wrote: At 2010-09-30 20:35, Trevor Parscal wrote: [...] * Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved. To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this: 1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code... The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch. There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than error on line ##. It's not an assumption. I AM one of the main developers working on bugs reported by users on pl.wiki. And yes line number is very helpful if accurate. Also note that I was picturing a situation where I have to debug the minified code when I don't get the error in my browser(s) at hand. Usually the problem is some extension/plug-in, but I can't tell unless I investigate. And reading minified code at it's current state is impossible for me. How can you possibly resolve an issue if you can't reproduce it? Developers have access to multiple browsers for this exact reason. Otherwise, you are debugging and programming blind, which is extremely unproductive. I'm very productive with inspecting code with my eyes. Ask my boss ;-). I understand the need to minify code I really do. I just don't get it why do you want to minify this that much. HTML which is usually served at every page load is not minified. Further more even minified HTML can be readable for debuging purposes as every debugger (inspector) shows the structure of the code rather then a code pulp. Also why are there are explicitly added comments like !-- /navigation --. People don't need comments and whitespace to read code, right? Also I have a PHP minifier if you are interested. Let's serve Mediawiki encode and if some developer want to debug the code he can always switch folders. I think that our outgoing HTML *should* be minified, and those comments should be stripped, I just have yet to get to that as of yet. Go to www.google.com and view source. Tell me what you see. Why are we so different from them? They are taking advantage of aggressive minification, and we should too. To be honest, I can't quite wrap my head around why I am getting any friction on making the front-end as fast as possible. I was expecting people to say I had not gone far enough. Two reasons probably: 1. The gain is not big for average user (loading big JS is only painful at first visit in a day or even more). 2. Wikipedia is not Google (you go to a Google's website and use it; you go to Wikipedia and change it). Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too). This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code. Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something
Re: [Wikitech-l] ResourceLoader Debug Mode
Trevor Parscal wrote: I think that our outgoing HTML *should* be minified, and those comments should be stripped, I just have yet to get to that as of yet. Go to www.google.com and view source. Tell me what you see. Why are we so different from them? They are taking advantage of aggressive minification, and we should too. Yes, I've looked at www.google.com's page source. It doesn't have a closing body tag or a closing html tag. It's also largely unreadable. I'm not sure why this is a something to strive for. Bandwidth on Wikimedia's end is cheap. The emerging bandwidth issues come largely from mobile platforms, which have separate mobile sites to address this. That isn't to say bandwidth should be needlessly wasted, but there's no reason to be paranoid about it. Whether or not you remove every newline from the HTML output is going to have a negligible impact for mobile and non-mobile platforms. It will have a demonstrable impact on developers and users, though, which I'll discuss below. To be honest, I can't quite wrap my head around why I am getting any friction on making the front-end as fast as possible. I was expecting people to say I had not gone far enough. You're getting friction because of three different issues. The first issue is that the front-end is currently pretty damn fast for _most_ people. It's serving cached HTML most of the time with very little JavaScript executed for a huge majority of users because (1) these users are not editing, they're just reading; and (2) these users are not logged in. I imagine there would be a lot more support for making, for example, page rendering faster. Anyone who has edited a large article logged in knows how painful it is. If ResourceLoader is going to address _that_ problem, then this is a marketing and public relations issue. The second issue is that you're focusing on the mess that was of your creation. There's a huge effort to clean up the insane amount of JavaScript that the UsabilityInitiative introduced. There isn't going to be much appreciation for cleaning up your own mess. Don't take that the wrong way, please. I'm not saying that the UsabilityInitiative's work wasn't good, but it came at a cost that now has to be addressed. You're asking why people aren't too excited about the clean-up and I'm offering my theory. The third issue is that, unlike www.google.com, Wikimedia wikis are editable sites. People customize their experience via gadgets, user scripts, and other things of that nature. The same isn't true for Google's homepage. The interactivity of Wikimedia wikis means that users need to be able to read the page source easily. This includes the HTML page source, the CSS, and the JavaScript that's loaded. Minification and obfuscation come at a cost when trying to develop scripts that modify the user experience. People have been trying to say this to you in so many ways, but you don't seem to be quite getting that. Yes, there's a debug mode via a URL parameter, but anyone who has ever dealt with settings set temporarily by URL parameter (?useskin, ?uselang, etc.) know what a pain in the ass these are. And, as with every feature ever introduced, there _will_ be bugs that come from minification and obfuscation. There doesn't seem to be an acknowledgement of this in a lot of your writing, which seems to be frustrating a lot of people. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode ( the need for a platform)
On 9/30/10 6:04 PM, MZMcBride wrote: You're getting friction because of three different issues. The first issue is that the front-end is currently pretty damn fast for _most_ people. The front end that we have *now* is fast for some people. The point of writing a new platform is to be able to do more, if we want. The second issue is that you're focusing on the mess that was of your creation. There's a huge effort to clean up the insane amount of JavaScript that the UsabilityInitiative introduced. I think we have a large difference of vision here. By contemporary standards the amount of JS in the UsabilityInitiative, and related projects, is a mere trifle. Our competitors invest a lot in Javascript infrastructure because they find that every last efficiency is usually worth it. If stripping newlines gives us 1K more per page, we can squeeze 1K more of usability or other interface niceties in. Now, we might decide not to do certain kinds of minification for other very good reasons. But I say this to put the Resource Loader in context. The point here, at least as I see it, is to bring Wikimedia sites up to the standards of the rest of the web, not to clean up some mess. The third issue is that, unlike www.google.com, Wikimedia wikis are editable sites. People customize their experience via gadgets, user scripts, and other things of that nature. The same isn't true for Google's homepage. I agree this is a very good reason why we should hesitate before doing anything that would obfuscate our pages. View Source doesn't work as well on the web any more, and it should on Wikipedia. (I personally would set the balance at a different point -- I'd like there to be a note right in the page source explaining how to view an unoptimized version.) That said, your assertion that Google doesn't allow customizations of its page is just not true. Google has offered skins for something like a year or more now, and their home page widget platform is literally called Gadgets. http://www.google.com/ig/ The great grandaddy of all site-Javascript modifiers is GreaseMonkey, and the guy that wrote that is even a Google employee (Aaron Boodman). Mihai Parparita, another Google employee, circulates a very popular set of Greasemonkey improvements to Gmail. So, the argument that Google etc. just don't get developer hacks, or that they do them badly, is quite wrong. Their platforms for doing this are far, far more successful than ours. We should be learning from them on this, not vice versa. Here's how they do it: Google guarantees that certain Javascript functions or DOM objects are reliably available for use and modification, and others are not. So they use minification for efficiency while explicitly preventing some things from being altered (that's a feature of Google's Closure Compiler). This is a story as old as computing; to enable a community, create a rich platform with a stable public API. Merely leaving the pages open to any modification is actually worse for gadget writers, since anything could change. Anyway, I think that's somewhat in the flavor of what the Resource Loader people are trying to achieve here. Efficiency *and* community. Minification *and* openness. Gadget-writers are a big part of their targeted use cases. Otherwise, they would have just used something off the shelf. There are a lot of good JS libraries out there, but none that quite fit the needs of our community. -- Neil Kandalgaonkar (| ne...@wikimedia.org ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
On 01/10/10 07:27, Roan Kattouw wrote: Minified code does not contain zero line breaks. Some line breaks are left in, although I don't understand exactly what the logic is there. The logic is that line breaks can have functional significance in two cases. One is where statement-ending semicolons are omitted: foo() bar() A semicolon is implied where the result of joining the lines would not be valid JS, in this case: foo() bar() This is not valid so a semicolon is implicitly added. The other case is where a line break acts like whitespace: x- -y Here the first - is the subtraction operator, and the second - is the unary negation operator. This is the same as x- -y but not the same as x--y, which is a syntax error since the -- is counted as the unary postfix decrement operator. Determining all of these cases precisely would be difficult, so JSMin just leaves the line break in where it follows any potentially problematic token. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode ( the need for a platform)
On 9/30/2010 10:03 PM, Neil Kandalgaonkar wrote: Here's how they do it: Google guarantees that certain Javascript functions or DOM objects are reliably available for use and modification, and others are not. So they use minification for efficiency while explicitly preventing some things from being altered (that's a feature of Google's Closure Compiler). This is a story as old as computing; to enable a community, create a rich platform with a stable public API. As a gadget developer, I would love that. However, that seems to be, in part, the opposite of what's happening. We're taking a public API that's been used for years (wikibits.js) and deprecating the whole thing, with no documentation (that I can find in a couple minutes of searching) for the replacements. When the new skin/editor was rolled out, there was basically no documentation for anything and there still isn't very much. When I rewrote the reftoolbar gadget[1] to support the new toolbar, in advance of the rollout, I had to do it by reading the Wikieditor code and testing on a personal sandbox wiki running trunk, which changed often and was frequently broken. Now I'm probably going to have to do that again to make sure it won't break with ResourceLoader. That's not a very good development process. Merely leaving the pages open to any modification is actually worse for gadget writers, since anything could change. Anyway, I think that's somewhat in the flavor of what the Resource Loader people are trying to achieve here. Efficiency *and* community. Minification *and* openness. Gadget-writers are a big part of their targeted use cases. Otherwise, they would have just used something off the shelf. There are a lot of good JS libraries out there, but none that quite fit the needs of our community. That's all great, but if you're going to go that route, you have to make damn sure you get everything right, else we're going to end up with a bunch of frustrated gadget developers and a ton of angry users with broken gadgets. [1] http://en.wikipedia.org/wiki/Wikipedia:RefToolbar_2.0 -- Alex (wikipedia:en:User:Mr.Z-man) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode ( the need for a platform)
Neil Kandalgaonkar wrote: The front end that we have *now* is fast for some people. The point of writing a new platform is to be able to do more, if we want. Do more, such as? If there's a broader outlook here, a roadmap for the future, what is it and where is it posted? I think there's a reasonable expectation that if you're going to make claims like this and push forward on projects like ResourceLoader, there be a list of all the gains that are going to be made as a result of it, or at least a list of all the foreseeable gains from it. You're talking about doing more. Be specific. I've looked at http://www.mediawiki.org/wiki/ResourceLoader and Trevor has said that the non-public roadmap will soon be public, though it's my understanding that the roadmap is more about deployment, not about the grand vision of what ResourceLoader will enable in the future. Can you shed some light on this? The second issue is that you're focusing on the mess that was of your creation. There's a huge effort to clean up the insane amount of JavaScript that the UsabilityInitiative introduced. I think we have a large difference of vision here. By contemporary standards the amount of JS in the UsabilityInitiative, and related projects, is a mere trifle. Our competitors invest a lot in Javascript infrastructure because they find that every last efficiency is usually worth it. If stripping newlines gives us 1K more per page, we can squeeze 1K more of usability or other interface niceties in. Which competitors are you referring to here? You seem to be suggesting that a lot of other sites bog down their users' experience with an obscene amount of JavaScript, so it's okay for Wikimedia to do the same, albeit with a bit less JavaScript. This doesn't follow. You also seem to be creating a mostly false trade-off through the idea of premature optimization. There isn't a fixed size a page can be. Making the HTML output unreadable and adding a feature aren't mutually exclusive. To suggest that they are isn't fair. The third issue is that, unlike www.google.com, Wikimedia wikis are editable sites. People customize their experience via gadgets, user scripts, and other things of that nature. The same isn't true for Google's homepage. I agree this is a very good reason why we should hesitate before doing anything that would obfuscate our pages. View Source doesn't work as well on the web any more, and it should on Wikipedia. (I personally would set the balance at a different point -- I'd like there to be a note right in the page source explaining how to view an unoptimized version.) That said, your assertion that Google doesn't allow customizations of its page is just not true. Google has offered skins for something like a year or more now, and their home page widget platform is literally called Gadgets. http://www.google.com/ig/ I was talking about www.google.com, as was the post I was replying to. Google has iGoogle, that much is certain, but I don't see how that's relevant to the broader discussion, sharing the use of the term gadgets notwithstanding. I think it's possible that your answer to the what are the future benefits of ResourceLoader? question might make what you were trying to say here clearer. Anyway, I think that's somewhat in the flavor of what the Resource Loader people are trying to achieve here. Efficiency *and* community. Minification *and* openness. Gadget-writers are a big part of their targeted use cases. Otherwise, they would have just used something off the shelf. There are a lot of good JS libraries out there, but none that quite fit the needs of our community. The broad point of my previous reply was that minification and obfuscation come at a real cost and the acknowledgement of this cost seems to be non-existent from the people pushing ResourceLoader forward. That's my take from reading the past discussion on this list, but perhaps I'm completely off the mark. When people talk about friction but can't understand the source of this friction, that indicates a problem to me. And it seems to be directly incompatible with your stated goal of efficiency *and* community. MZMcBride ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
On 01/10/10 04:35, Trevor Parscal wrote: OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume. I took this graph: http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for internet hosting, of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month. I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case. I don't know what that kind of bandwidth costs the foundation, but it's not free. Developer time is not free either. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
I was hardly making a case for how amazingly expensive it was. I was running some basic calculations that seemed to support your concept of fairly cheap, but chose to mention that it's still not free. - Trevor On 9/30/10 9:30 PM, Tim Starling wrote: On 01/10/10 04:35, Trevor Parscal wrote: OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume. I took this graph: http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for internet hosting, of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month. I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case. I don't know what that kind of bandwidth costs the foundation, but it's not free. Developer time is not free either. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
This is getting a little out of hand? People are going to spend more time talking about the potential for minification errors or sub optimisation cost criteria then we will ever actually be running into real minification errors or any real readability issue. Reasonable efforts are being made to make the non-minified versions easily accessible. We can add additional comment to every outputted file that says request me with ?debug=true to get the source code version, add a user preference, add a cookie, and a view non-minfied source link to the bottom every page... But try to keep in mind the whole point of minification is identical program execution while minimising the file size! If we are to optimise with random adjustments for readability we are optimising in two opposite directions, and every enhacment in the direction of optimized pacakge code delivery could potentially go against the 'readability' optimisation. We are already commited to supporting two modes one that optimized readability with raw source code delivery and one that is optimised for small packaged delivery. No sense in setting readability as criteria for the packaged mode since the 'readability' mode will always do 'readability' better. --michael On 10/01/2010 12:38 AM, Trevor Parscal wrote: I was hardly making a case for how amazingly expensive it was. I was running some basic calculations that seemed to support your concept of fairly cheap, but chose to mention that it's still not free. - Trevor On 9/30/10 9:30 PM, Tim Starling wrote: On 01/10/10 04:35, Trevor Parscal wrote: OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume. I took this graph: http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for internet hosting, of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month. I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case. I don't know what that kind of bandwidth costs the foundation, but it's not free. Developer time is not free either. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
On Thu, Sep 30, 2010 at 8:27 AM, Trevor Parscal tpars...@wikimedia.org wrote: ... When debug mode is off: * Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minified When debug mode is on: * Modules are requested individually * Resources are combined into modules There are two features here. modules optimising traffic It seems everyone realises that the serving of raw files has to remain. Why is this old and remaining functionality going to be named 'debug mode'? Why not give the new feature a name, put it in the prefs, and turn that pref on by default after sufficient road testing. As happened with the new skin. There are going to be mediawiki implementers and wikimedia users who want to disable this feature for reasons other than debugging. (I've always considered the 'Misc' section of the prefs to be 'Performance') -- John Vandenberg ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] ResourceLoader Debug Mode
Thank you. +1. - Trevor On 9/30/10 10:04 PM, Michael Dale wrote: This is getting a little out of hand? People are going to spend more time talking about the potential for minification errors or sub optimisation cost criteria then we will ever actually be running into real minification errors or any real readability issue. Reasonable efforts are being made to make the non-minified versions easily accessible. We can add additional comment to every outputted file that says request me with ?debug=true to get the source code version, add a user preference, add a cookie, and a view non-minfied source link to the bottom every page... But try to keep in mind the whole point of minification is identical program execution while minimising the file size! If we are to optimise with random adjustments for readability we are optimising in two opposite directions, and every enhacment in the direction of optimized pacakge code delivery could potentially go against the 'readability' optimisation. We are already commited to supporting two modes one that optimized readability with raw source code delivery and one that is optimised for small packaged delivery. No sense in setting readability as criteria for the packaged mode since the 'readability' mode will always do 'readability' better. --michael On 10/01/2010 12:38 AM, Trevor Parscal wrote: I was hardly making a case for how amazingly expensive it was. I was running some basic calculations that seemed to support your concept of fairly cheap, but chose to mention that it's still not free. - Trevor On 9/30/10 9:30 PM, Tim Starling wrote: On 01/10/10 04:35, Trevor Parscal wrote: OK, now I've calculated it... On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference. While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume. I took this graph: http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for internet hosting, of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month. I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case. I don't know what that kind of bandwidth costs the foundation, but it's not free. Developer time is not free either. -- Tim Starling ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
