Re: [Catalyst] Backlog for proposed changes in next Catalyst release
so I don't really see the point of adding extra decoding and encoding all over the place Because it's security relevant. By now the article http://perlmonks.org/?node_id=644786 should be well-known. Yes, this specific circumstance shown there is difficult to trigger and exploit. That's not the point. One must be in the correct mindset that even character encoding can be an attack vector. As a Perl programmer, you must be aware of the difference between UTF8 and UTF-8 and how decoding at the perimeter (instead of passing through, as you described) is beneficial. signature.asc Description: PGP signature ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
While it's not Catalyst's fault, I've found over the years that interacting with underlying libraries, databases and legacy systems is generally easier when I *don't* try to force anything. I have custom code in place to deal with know sources of inconsistent encodings (check to see if it's valid UTF8, look up the remainder in a table painstakingly assembled over a period of time that catches a few odd MacRoman characters that show up in some of our contributors' data, fall back to latin1 or cp1252 for the remainder, leave anything else as \xNN). Everywhere else, UTF8 can be passed through quite transparently, so I don't really see the point of adding extra decoding and encoding all over the place to switch from utf8, to some internal wide character encoding, then back to utf8 again for output. One of the positive features of UTF8 has always been that code that doesn't need to identify any of those fancy accented characters can just treat it the same as ASCII, Latin-$WHATEVER or cp1252 without any overhead. Overall I can't see the point of forcing everything to be converted multiple times ... On 3 March 2013 16:50, Bill Moseley mose...@hank.org wrote: On Fri, Mar 1, 2013 at 9:38 AM, John Napiorkowski jjn1...@yahoo.comwrote: Hey All, http://jjnapiorkowski.typepad.com/modern-perl/2013/03/catalyst-backup-for-next-release-on-play-perl.html This is over on play Perl (http://play-perl.org/player/jnap) for your comments and votes. Hope to see you there! I think I've asked this before, but is there any talk of native encoding support -- meaning make Catalyst::Plugin::Unicode::Encoding part of the framework? Having it a plugin makes it appear as optional, but the correct approach is to decoded all text requests and encode all text responses. This is fresh in my mind because last week had problems with two separate encoding issues. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On 10 March 2013 19:58, Tomas Doran bobtf...@bobtfish.net wrote: Does anyone have a strong opinion on this being added to their app by default? Does anyone have an app which needs you to NOT load the unicode plugin? Speak now, or you're gonna have a bad time :) Think I kind of meant to reply to this message just now, not the thread starter :( ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
From: Bill Moseley mose...@hank.org To: The elegant MVC web framework catalyst@lists.scsys.co.uk Sent: Monday, March 11, 2013 11:48 AM Subject: Re: [Catalyst] Backlog for proposed changes in next Catalyst release On Mon, Mar 11, 2013 at 4:53 AM, Will Crawford billcrawford1...@gmail.com wrote: While it's not Catalyst's fault, I've found over the years that interacting with underlying libraries, databases and legacy systems is generally easier when I *don't* try to force anything. I have custom code in place to deal with know sources of inconsistent encodings (check to see if it's valid UTF8, look up the remainder in a table painstakingly assembled over a period of time that catches a few odd MacRoman characters that show up in some of our contributors' data, fall back to latin1 or cp1252 for the remainder, leave anything else as \xNN). Everywhere else, UTF8 can be passed through quite transparently, so I don't really see the point of adding extra decoding and encoding all over the place to switch from utf8, to some internal wide character encoding, then back to utf8 again for output. One of the positive features of UTF8 has always been that code that doesn't need to identify any of those fancy accented characters can just treat it the same as ASCII, Latin-$WHATEVER or cp1252 without any overhead. Overall I can't see the point of forcing everything to be converted multiple times ... I think we can all agree that historically encoding has been confusing, misunderstood, and frequently ignored. And very often just done plain wrong. This is exactly the problem. Part of it I think is how some many companies have trouble installing stuff from CPAN, so that don't and or don't upgrade very much. Although this is a problem we as a community need to help with, I think this is part of why I am pushing to core the obvious stuff into Catalyst, such as to make it work more productively and correctly out of the box. [SNIP] I have never considered any performance aspect of this. It never shows up when we profile slow responses. Plus, it's never been an optional operation. We manipulate characters and we exchange data as bytes. You have to convert between those. The plugin should be core to Catalyst. It think it's pretty safe to add it if it only encodes if the utf8 flag is set on the body -- that should prevent double-encodings. And having a config option to disable is easy. And if the plugin is found on the app issue a warning. It's possible that someone has their own modified version of the plugin using the same name. I think it would be safe to core the behavior but have it off by default for the existing legacy project skeleton generator. No doubt it might cause trouble for some people that have hacking Catalyt.pm a lot, but I'm going to go out on a limb and suggest if you've hack core you will be able to fix it. And if not, then you shouldn't have hacked it ;) The 5.9 series will continue to err today backcompat for a while, although I would like to lay the groundwork for the features we'd like in Catalyst 6, whichi I think we all agree would be allowed to break some backward compat in the name of improving the codebase and setting the stage for future improvements. My thinking here is that 5.9xxx would remain hacked on for security and bug fixes only, and I really hope that companies that maintain that version would step up a bit to help. At a certain point I think companies can't expect volunteers to continue basically working for them for free ;) -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
Turning it into a role and making it optionally be applied would be a nice to have (i.e. so people could turn it off if needed). It working seamlessly with apps which already use the (old) plugin is a *must* have (but to be fair - maybe we can just filter U::P::U::Encoding out of the plugin list if there, and break hard if old C::P::Unicode is found?) Does anyone have a strong opinion on this being added to their app by default? Does anyone have an app which needs you to NOT load the unicode plugin? Speak now, or you're gonna have a bad time :) I'm all for this, and I'm feeling that our strategy of doing obsessive backwards compatibility has hurt us in the past. If you're not doing unicode right, your app is _broken_, and you need to fix it - if that stops you upgrading, then so be it. If anyone cares enough to say this should make it 6.0, that's also cool. Catalyst 6.0 mandatory being correct… Job done! Cheers t0m On 5 Mar 2013, at 15:42, Bill Moseley mose...@hank.org wrote: On Tue, Mar 5, 2013 at 7:09 AM, John Napiorkowski jjn1...@yahoo.com wrote: Bill, I think there's general agreement that the plugin is stable enough and right enough that it should represent core behavior. Although one of Catalyst's strengths has been to be un-opinionated and leave stuff to external distributions when possible, I think this does fall under the 'all good apps should work this way'. I've tentatively pegged it for the March Catalyst release, since it seems pretty straightforward, at least for now. If you think you can fit it in (or talk your boss/clients/whatever into paying for your time) please give a shout out. It would be great to see more people get to know Catalyst internals a bit, otherwise I don't really see how we are going to be able to move the platform forward. The month of May is more likely for me, unfortunately. I was thinking of turning the plugin into a Role and then consuming if the plugin was not already listed or if a config option to disable was not set. That is, on by default but can be disabled. Is that in line with what you were thinking? Ha! I was looking at the code just now and saw a comment that included this URL: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg02350.html I guess I've thought about this before I wonder what percent of Catalyst apps make use of that plugin. Not sure, but I think the play-perl ticket has a good way to not break most people's code -- Bill Moseley mose...@hank.org -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On 10/03/13 19:58, Tomas Doran wrote: Does anyone have a strong opinion on this being added to their app by default? Does anyone have an app which needs you to NOT load the unicode plugin? Speak now, or you're gonna have a bad time :) My views already encode the response body if required. If C::P::U::Encoding::finalize_headers (which I don't use) moves to Catalyst core then my HTML response body will be encoded twice. You need to make these changes optional. Also, the line # Only touch 'text-like' contents return $c-next::method(@_) unless $c-response-content_type =~ /^text|xml$|javascript$/; smells bad, it will apply to text/x-json but not application/json -- Regards ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Sun, Mar 10, 2013 at 2:06 PM, Peter Flanigan p...@roxsoft.co.uk wrote: # Only touch 'text-like' contents return $c-next::method(@_) unless $c-response-content_type =~ /^text|xml$|javascript$/; smells bad, it will apply to text/x-json but not application/json The code that I use only encodes if the utf8 flag is set. The JSON encoders (at least what I use) encode to utf8, so my code doesn't touch JSON as it's already been encoded. I decode all input -- database, templates, text files, web requests and API requests. All data that represents characters must be decoded on input. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
From: Bill Moseley mose...@hank.org To: John Napiorkowski jjn1...@yahoo.com; The elegant MVC web framework catalyst@lists.scsys.co.uk Sent: Tuesday, March 5, 2013 12:39 AM Subject: Re: [Catalyst] Backlog for proposed changes in next Catalyst release On Mon, Mar 4, 2013 at 9:24 AM, John Napiorkowski jjn1...@yahoo.com wrote: http://play-perl.org/quest/5134cdc18e0a96450f38 There was a discussion about this on #catalyst-dev and in general people seem to think its ok. Bill, are you up to giving the work a try? I'd be happy to mentor you. I don't think its killer work, but would really help I would like to give it a try, yes. I'm just not sure when. I guess there's no rush since there's a plugin. I was just thinking it would be wise to make it a standard part of the framework since it's something that every app probably should to, but easy to ignore or get wrong. Bill, I think there's general agreement that the plugin is stable enough and right enough that it should represent core behavior. Although one of Catalyst's strengths has been to be un-opinionated and leave stuff to external distributions when possible, I think this does fall under the 'all good apps should work this way'. I've tentatively pegged it for the March Catalyst release, since it seems pretty straightforward, at least for now. If you think you can fit it in (or talk your boss/clients/whatever into paying for your time) please give a shout out. It would be great to see more people get to know Catalyst internals a bit, otherwise I don't really see how we are going to be able to move the platform forward. I wonder what percent of Catalyst apps make use of that plugin. Not sure, but I think the play-perl ticket has a good way to not break most people's code -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Tue, Mar 5, 2013 at 7:09 AM, John Napiorkowski jjn1...@yahoo.com wrote: Bill, I think there's general agreement that the plugin is stable enough and right enough that it should represent core behavior. Although one of Catalyst's strengths has been to be un-opinionated and leave stuff to external distributions when possible, I think this does fall under the 'all good apps should work this way'. I've tentatively pegged it for the March Catalyst release, since it seems pretty straightforward, at least for now. If you think you can fit it in (or talk your boss/clients/whatever into paying for your time) please give a shout out. It would be great to see more people get to know Catalyst internals a bit, otherwise I don't really see how we are going to be able to move the platform forward. The month of May is more likely for me, unfortunately. I was thinking of turning the plugin into a Role and then consuming if the plugin was not already listed or if a config option to disable was not set. That is, on by default but can be disabled. Is that in line with what you were thinking? Ha! I was looking at the code just now and saw a comment that included this URL: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg02350.html I guess I've thought about this before I wonder what percent of Catalyst apps make use of that plugin. Not sure, but I think the play-perl ticket has a good way to not break most people's code -- Bill Moseley mose...@hank.org -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
From: Bill Moseley mose...@hank.org To: John Napiorkowski jjn1...@yahoo.com Cc: The elegant MVC web framework catalyst@lists.scsys.co.uk Sent: Tuesday, March 5, 2013 10:42 AM Subject: Re: [Catalyst] Backlog for proposed changes in next Catalyst release On Tue, Mar 5, 2013 at 7:09 AM, John Napiorkowski jjn1...@yahoo.com wrote: Bill, I think there's general agreement that the plugin is stable enough and right enough that it should represent core behavior. Although one of Catalyst's strengths has been to be un-opinionated and leave stuff to external distributions when possible, I think this does fall under the 'all good apps should work this way'. I've tentatively pegged it for the March Catalyst release, since it seems pretty straightforward, at least for now. If you think you can fit it in (or talk your boss/clients/whatever into paying for your time) please give a shout out. It would be great to see more people get to know Catalyst internals a bit, otherwise I don't really see how we are going to be able to move the platform forward. The month of May is more likely for me, unfortunately. I was thinking of turning the plugin into a Role and then consuming if the plugin was not already listed or if a config option to disable was not set. That is, on by default but can be disabled. Is that in line with what you were thinking? There's a quest on playperl about it http://play-perl.org/quest/5134cdc18e0a96450f38 which has the general outline. I think this could be done in a few hours, definitely less than a day, unless something weird shows up in the test cases (I'm guessing some tests might pointlessly fail and need updating. Its mostly legwork john Ha! I was looking at the code just now and saw a comment that included this URL: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg02350.html I guess I've thought about this before I wonder what percent of Catalyst apps make use of that plugin. Not sure, but I think the play-perl ticket has a good way to not break most people's code -- Bill Moseley mose...@hank.org -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
Bill, I'd be open to it, if it seems like we always just need to add this. I'm not enough an expert of unicode to understand the drawbacks, or why this was done as a plugin in the first place. Is there a drawback, for example to having it core? I can see at the beginning having stuff like this as stand alone dists, since it lets you be more agile in fixing issues with it, and all that but if this is stable then I can see just making it core. Anyone? John From: Bill Moseley mose...@hank.org To: The elegant MVC web framework catalyst@lists.scsys.co.uk Sent: Sunday, March 3, 2013 11:50 AM Subject: Re: [Catalyst] Backlog for proposed changes in next Catalyst release On Fri, Mar 1, 2013 at 9:38 AM, John Napiorkowski jjn1...@yahoo.com wrote: Hey All, http://jjnapiorkowski.typepad.com/modern-perl/2013/03/catalyst-backup-for-next-release-on-play-perl.html This is over on play Perl (http://play-perl.org/player/jnap) for your comments and votes. Hope to see you there! I think I've asked this before, but is there any talk of native encoding support -- meaning make Catalyst::Plugin::Unicode::Encoding part of the framework? Having it a plugin makes it appear as optional, but the correct approach is to decoded all text requests and encode all text responses. This is fresh in my mind because last week had problems with two separate encoding issues. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Mar 4, 2013, at 11:12 AM, John Napiorkowski wrote: Bill, I'd be open to it, if it seems like we always just need to add this. I'm not enough an expert of unicode to understand the drawbacks, or why this was done as a plugin in the first place. Is there a drawback, for example to having it core? I can see at the beginning having stuff like this as stand alone dists, since it lets you be more agile in fixing issues with it, and all that but if this is stable then I can see just making it core. Correctly understanding and handling character encodings can be tricky. It's quite common to encounter applications that behave erratically when fed non-ASCII input (with a last name that contains such characters, I would know). That say, I cannot see how including that functionality in the core would be a bad thing. Best regards -lem ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
http://play-perl.org/quest/5134cdc18e0a96450f38 There was a discussion about this on #catalyst-dev and in general people seem to think its ok. Bill, are you up to giving the work a try? I'd be happy to mentor you. I don't think its killer work, but would really help John - Original Message - From: Luis E. Muñoz luisemu...@gmail.com To: John Napiorkowski jjn1...@yahoo.com; The elegant MVC web framework catalyst@lists.scsys.co.uk Cc: Sent: Monday, March 4, 2013 11:38 AM Subject: Re: [Catalyst] Backlog for proposed changes in next Catalyst release On Mar 4, 2013, at 11:12 AM, John Napiorkowski wrote: Bill, I'd be open to it, if it seems like we always just need to add this. I'm not enough an expert of unicode to understand the drawbacks, or why this was done as a plugin in the first place. Is there a drawback, for example to having it core? I can see at the beginning having stuff like this as stand alone dists, since it lets you be more agile in fixing issues with it, and all that but if this is stable then I can see just making it core. Correctly understanding and handling character encodings can be tricky. It's quite common to encounter applications that behave erratically when fed non-ASCII input (with a last name that contains such characters, I would know). That say, I cannot see how including that functionality in the core would be a bad thing. Best regards -lem ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Mon, Mar 4, 2013 at 9:24 AM, John Napiorkowski jjn1...@yahoo.com wrote: http://play-perl.org/quest/5134cdc18e0a96450f38 There was a discussion about this on #catalyst-dev and in general people seem to think its ok. Bill, are you up to giving the work a try? I'd be happy to mentor you. I don't think its killer work, but would really help I would like to give it a try, yes. I'm just not sure when. I guess there's no rush since there's a plugin. I was just thinking it would be wise to make it a standard part of the framework since it's something that every app probably should to, but easy to ignore or get wrong. I wonder what percent of Catalyst apps make use of that plugin. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Fri, Mar 1, 2013 at 9:38 AM, John Napiorkowski jjn1...@yahoo.com wrote: Hey All, http://jjnapiorkowski.typepad.com/modern-perl/2013/03/catalyst-backup-for-next-release-on-play-perl.html This is over on play Perl (http://play-perl.org/player/jnap) for your comments and votes. Hope to see you there! I think I've asked this before, but is there any talk of native encoding support -- meaning make Catalyst::Plugin::Unicode::Encoding part of the framework? Having it a plugin makes it appear as optional, but the correct approach is to decoded all text requests and encode all text responses. This is fresh in my mind because last week had problems with two separate encoding issues. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Backlog for proposed changes in next Catalyst release
Hey All, http://jjnapiorkowski.typepad.com/modern-perl/2013/03/catalyst-backup-for-next-release-on-play-perl.html This is over on play Perl (http://play-perl.org/player/jnap) for your comments and votes. Hope to see you there! John ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
