Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Anthony Lucas
I don't see how DBI can be corrupting your data. DBD::ODBC, or more specifically the actual driver it is using will be more suspect in that case. In either case, setting DBI's trace flag to something high should provide some insight who's doing what, and you can make sure they are being handled a

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Alexander Hartmaier
On 2013-07-02 19:26, Bill Moseley wrote: On Tue, Jul 2, 2013 at 9:29 AM, Craig Chant mailto:cr...@homeloanpartnership.com>> wrote: >> All the above seems overkill. I suspect what you want is closer to this: >> (but see notes below). Tried that, didn't work, ended up in a long Catalyst discus

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Bill Moseley
On Tue, Jul 2, 2013 at 9:29 AM, Craig Chant wrote: > >> All the above seems overkill. I suspect what you want is closer to > this: (but see notes below). > > > > Tried that, didn’t work, ended up in a long Catalyst discussion where it > was worked out that I needed to wrap any XLS output to an

Re: [Catalyst] How to change default LICENSE in app and files

2013-07-02 Thread bill hauck
Thanks for the quick reply. I'll see what I can come up with. Cheers > > From: Tomas Doran >To: bill hauck ; The elegant MVC web framework > >Sent: Tuesday, July 2, 2013 9:26 AM >Subject: Re: [Catalyst] How to change default LICENSE in app and files > > >Y

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
>> All the above seems overkill. I suspect what you want is closer to this: >> (but see notes below). Tried that, didn't work, ended up in a long Catalyst discussion where it was worked out that I needed to wrap any XLS output to an IO:FILE handle otherwise Catalyst dies with an "out of memor

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Bill Moseley
Before we get a long utf8 discussion here: On Tue, Jul 2, 2013 at 8:58 AM, Bill Moseley wrote: > > Personally, I think the correct approach is to only encode *character* data > -- that is check to see if the utf8 flag is set before calling encode. > I say that with the caveat that ALL textual i

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Bill Moseley
On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant wrote: > > # output header > $c->response->content_type('application/vnd.ms-excel'); > $c->response->content_length(length($xls)); > $c->response->header(Content_Disposition => > 'attachment;filename=NBCS_Export.csv'); > >

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
I looked at that link but it doesn't have anything on DBD::ODBC that I could see. Interesting enough utf8::is_utf8($xls) returns true, but it isn't because the output is garbage? Also as utf8::is_utf8 claims the string is UTF-8, why does decode baulk with 'cannot decode string with wide chara

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Mike Whitaker
On 2 Jul 2013, at 14:36, Mike Whitaker wrote: > On 2 Jul 2013, at 14:34, Craig Chant wrote: >> >> So either I refactor my SQL model wrappers under Catalyst to use >> Win32::ODBC, or I find out why DBI is corrupting my SQL data. > > Be a hero. Do the latter :D :D ..and have a read of http://ju

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Mike Whitaker
On 2 Jul 2013, at 14:34, Craig Chant wrote: > > So either I refactor my SQL model wrappers under Catalyst to use Win32::ODBC, > or I find out why DBI is corrupting my SQL data. Be a hero. Do the latter :D :D ___ List: Catalyst@lists.scsys.co.uk Listin

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
Hi Alexander, Well I think I found the problem... the DBI module! I went back to my original SQL wrapper for Win32::ODBC in my original procedural script application before porting to Catalyst (where I moved to DBI). All my output to XLS in the old script using Win32::ODBC works perfectly, no

Re: [Catalyst] How to change default LICENSE in app and files

2013-07-02 Thread Tomas Doran
You can't, sorry. Patch to be able to do so would be welcome :) Cheers t0m On 2 Jul 2013, at 02:46, bill hauck wrote: > Hi > > I was wondering if there's a way to change the default LICENSE that is added > to each Controller through myapp_create.pl. I'd like to use GNU Affero GPL > (http:

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Alexander Hartmaier
On 2013-07-02 14:20, Craig Chant wrote: > Hi Alexander, > > I have read the DBD::ODBC docs and it seems Unicode is ON by default > > " Enabling and Disabling Unicode support > > On Windows Unicode support is enabled by default" > > So do I assume the data coming from the DB is Unicode encoded alrea

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
Hi Alexander, I have read the DBD::ODBC docs and it seems Unicode is ON by default " Enabling and Disabling Unicode support On Windows Unicode support is enabled by default" So do I assume the data coming from the DB is Unicode encoded already? if so , why does decode fall over? I don't seem

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Alexander Hartmaier
On 2013-07-02 13:45, Craig Chant wrote: > Hi Mike, > > The data is coming from MS SQL 2008 R2, I have checked the DB and the column > is defined as data type nvarchar(max) , so is set to Unicode and when viewing > data via SQL Studio Manager, the Unicode characters display fine. > > I am accessin

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
Hi Mike, The data is coming from MS SQL 2008 R2, I have checked the DB and the column is defined as data type nvarchar(max) , so is set to Unicode and when viewing data via SQL Studio Manager, the Unicode characters display fine. I am accessing the DB via DBI and have set on the connection stri

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Mike Whitaker
> # encode UTF8 octet >use Encode qw(decode encode); >my $octets = encode('UTF-8', $xls); OUt of curiousity, where's the data in $xls come from, and what do you know about its encoding and bytes vs characters status? ___ List: Cat

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
well Lars said that binmode(":encoding...") wasn't encoding. I thought it was, I very confused here! Anyway, I've also tried... # encode UTF8 octet use Encode qw(decode encode); my $octets = encode('UTF-8', $xls); # output he

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Mike Whitaker
On 2 Jul 2013, at 10:59, Craig Chant wrote: > >utf8::encode($xls); > >$iof->binmode(":encoding(UTF-8)"); ...erm, haven't you now encoded twice? Also? Don't use utf8::encode - if you can't use IO layers etc, use the general Encode module. ___

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Mike Whitaker
On 2 Jul 2013, at 10:38, Craig Chant wrote: > > I thought all strings in perl were Unicode ? or at least that's what the > Unicode::Encoding plugin is for isnt' it? No. Decoded strings in Perl are in a character representation which is magic and you don't need to know about. Assuming that rep

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
To follow up having looked at perldoc http://perldoc.perl.org/utf8.html I have just tried... - # encode UTF8 use utf8; utf8::encode($xls); # output header $c->response->content_type('application/vnd.ms-excel'); $c->response->co

RE: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
Hi Lars, Thanks for the reply >> The problem is rather that Catalyst::Response::body wants octets Sorry, you've lost me, I thought response body wanted an IO::File Handle which is what I have given it? I was also under the impression that $c->response->body( 'Page not found' ); was accept

Re: [Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯
> I seem to be having major problems getting the application devel > server (my_app_server.pl) to accept an IO::File Handle as CSV and > correctly encode as UTF-8. No, the code does exactly what you said. The problem is rather that Catalyst::Response::body wants octets, not characters. Either pass

[Catalyst] CSV / UTF-8 / Unicode

2013-07-02 Thread Craig Chant
Hi, I seem to be having major problems getting the application devel server (my_app_server.pl) to accept an IO::File Handle as CSV and correctly encode as UTF-8. I have Unicode::Encoding in my_app.pm and am using the following to wrap up the CSV string. # create an IO::File for Cata