RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Nice one Mike spot on!

I did a bit of digging myself before trying anything and found this blog : 
http://blogs.msdn.com/b/qingsongyao/archive/2009/04/10/sql-server-and-utf-8-encoding-1-true-or-false.aspx

You learn something new every day!

It seems MS encoding is 'UCS-2' which is equivalent to 'UTF-16' (sort of, but 
not exact!)

Perhaps Win32::ODBC handles this natively whereas DBI::DBD::ODBC doesn't?

Having read the UCS-2 info at the bottom of the DBD::ODBC CPAN docs it seems 
DBD::ODBC is messy as hell trying to support UCS-2 / UTF-16.

I have tried within Perl to use decode against the string from the DBI fetch, 
but it just errors...

[error] Caught exception in My_App::Controller:: XLSExport->export "Wide 
character in subroutine entry at C:/Perl/lib/Encode.pm line 176."

And that was using UTF-16 and UCS-2 / UCS-2BE encoding types, is seems as per 
the encode docs "Bogus surrogates result in death."

And I guess as utf8::is_utf8 comes back as true and the fact it states on the 
DBD::ODBC docs...

" Perl scalars which are UTF-8 and are sent through the ODBC API will be 
converted to UTF-16 and passed to the ODBC wide APIs or signalled as SQL_WCHARs 
(e.g., in the case of bound columns). Retrieved data which are wide characters 
are converted from UTF-16 to UTF-8. However, you should realise most ODBC 
drivers do not support UTF-16"

It seems the DBD::ODBC driver is auto converting the data from UTF-16 to UTF-8, 
and that is where the corruption must be occurring, it even states that UTF-16 
isn't really supported under ODBC?

So what I have is UTF-8, but after a conversion by the ODBC driver from UCS-2 
using UTF-16 encoding, which appears to be making a mess of things.

I have checked and I was running DBI v 1.616, so installed latest version 1.627 
plus upgraded DBD::ODBC to 1.43 but exactly the same problem!

I'm at a loss what else I can do other than throw DBI away and refactor to 
Win32::ODBC , it seems DBI::DBD::ODBC is converting the data on retrieval into 
UTF-8 and making a pigs ear of it! I can't see any simple switch or parameter 
to tell DBD::ODBC to do things differently, can you?

There is a whole bunch of Unicode maybe's and caveats on the DBD::ODBC CPAN 
docs, yet if you scan the Win32::ODBC CPAN docs for a mention of UTF-8 / UTF-16 
/ UCS-2 or the word Unicode, none of them are mentioned even once, so is that a 
significant indicator of something?

I've checked with DBD::ODBC and Win32::ODBC , that the ODBC.pm is not the same 
module being used, I can see a flag in the DBI ODBC.pm module of 
"odbc_has_unicode", but it is an read only  attribute.

Can you suggest anything else that might prevent DBI from corrupting the data 
on UTF-8 conversion?

Thanks,
Craig.








-Original Message-
From: Mike Whitaker [mailto:m...@altrion.org]
Sent: 04 July 2013 10:35
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

On 4 Jul 2013, at 10:30, Mike Whitaker  wrote:
> On 4 Jul 2013, at 09:56, Craig Chant  wrote:
>>
>> Yes it's NVARCHAR(max) , which I understood is MS's data-typing for uNicode 
>> VARiable CHARacters, looking at some sample column data via the Windows SQL 
>> Management GUI, it appears to display ok.
>
> It probably isn't UTF-8, though. UTF-8 is only one possible encoding in which 
> you can store unicode character points.

Try, on the off-chance I've read the spec right

- not bothering with mysql_encode_utf8 in the DBI connect args
- passing all data from DBI through decode("UTF-16",...) or decode("UCS-2"...) 
- MS's docs aren't that clear which!
- reencoding it as utf8 on the way out.



___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communicati

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

On 4 Jul 2013, at 10:30, Mike Whitaker  wrote:
> On 4 Jul 2013, at 09:56, Craig Chant  wrote:
>> 
>> Yes it's NVARCHAR(max) , which I understood is MS's data-typing for uNicode 
>> VARiable CHARacters, looking at some sample column data via the Windows SQL 
>> Management GUI, it appears to display ok.
> 
> It probably isn't UTF-8, though. UTF-8 is only one possible encoding in which 
> you can store unicode character points.

Try, on the off-chance I've read the spec right

- not bothering with mysql_encode_utf8 in the DBI connect args
- passing all data from DBI through decode("UTF-16",...) or decode("UCS-2"...) 
- MS's docs aren't that clear which!
- reencoding it as utf8 on the way out.



___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

On 4 Jul 2013, at 09:56, Craig Chant  wrote:
> 
> Yes it's NVARCHAR(max) , which I understood is MS's data-typing for uNicode 
> VARiable CHARacters, looking at some sample column data via the Windows SQL 
> Management GUI, it appears to display ok.

It probably isn't UTF-8, though. UTF-8 is only one possible encoding in which 
you can store unicode character points.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Thanks Anthony,



I will try your suggestions when I'm back in the office tomorrow, I'm on a 
study day today for my OU course.



Would be good to track this down and fix without having to refactor to 
Win32::ODBC , I eventually want to look at replacing my own DBI wrapper for 
DBIC ORM but am concerned this wouldn't be possible if I cant get DBI to play 
ball with MS.



>> Your data is being stored in Unicode data typed columns right?



Yes it's NVARCHAR(max) , which I understood is MS's data-typing for uNicode 
VARiable CHARacters, looking at some sample column data via the Windows SQL 
Management GUI, it appears to display ok.



I know that the data being pasted into it is coming from an MS Access front end 
application that is linked to the same backend SQL server.



I also know that this is a memo / rich text input box control on the form 
(view) bound directly to the table column via a linked table definition with 
the backend SQL server and some of what they enter they copy/paste from emails 
and MS Word documents (and possibly PDF)



I can't see any odd characters looking at a small amount of sample data on the 
SQL server, and the data comes out of Win32::ODBC looking ok too.



>From what I can tell the data is in Unicode during capture and storage, it's 
>just the retrieval with DBI  where it seems to be breaking down.



I have to include a longread setting when using DBI::DBD::ODBC with SQL  
already, otherwise it falls over with the data being to long, so perhaps there 
is another parameter I need?



I really appreciate all the help you guys have given so far, thank you.



Regards,



Craig






From: Anthony Lucas [anthonyjlu...@gmail.com]
Sent: 04 July 2013 01:09
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode


On 3 July 2013 11:18, Craig Chant 
mailto:cr...@homeloanpartnership.com>> wrote:

>> Maybe write a standalone test and take Catalyst and browser quirks out of 
>> the picture.

I have already done this, I have two SQL wrapper modules one that uses 
DBI::DBD::ODBC and one that uses Win32::ODBC, I applied it to the same 
standalone script that produces CSV output, the only difference between the 
test was one test accessed SQL with the DBI SQL wrapper and one test accessed 
SQL with the Win32::ODBC SQL wrapper, DBI outputted junk chars, Win32::ODBC 
didn't. What else should I be doing to test for the culprit of the corruption?

You need to see how they are using the ODBC API underneath for handling the 
data and encoding.
Setting the trace flag on DBI (i.e. DBI->trace(n)) will expose the DBD::ODBC 
activity. I'm not sure of the debugging available for Win32::ODBC.

One thing I would check first is what they are treating the column data as. If 
DBD::ODBC is treating the columns as WCHAR but Win32::ODBC is treating them as 
CHAR and then doing extra "magic" decoding (or not), well then you've found a 
big clue. There has to be different handling or differing levels of ODBC 
support somewhere.

I would assume that DBD::ODBC is doing "the right thing", and something else is 
amiss upstream (but well, never assume with Unicode handling, so make sure with 
the trace).



>> Also, you are aware that your data will probably be coming back as UCS2 if 
>> you're using SQL Server right?

No, what is UCS2 and is this handled differently in DBI::DBD::ODBC vs 
Win32::ODBC ?


>From what I understand, is ultimately what you've got happening?:
Original Input Data -> SQL Client -> Database Driver -> Database (UCS2) -> 
Windows ODBC Driver -> DBD::ODBC -> Catalyst(?)

If so, since you're storing the data as Unicode and the database driver knows 
this (because your column type is NVARCHAR etc.), conversion to UCS2 happens at 
the driver stage on Windows. This is lossless between the different Unicodes, 
so just make sure your input is actual good Unicode up to that point and your 
data is being stored correctly.

Your data is being stored in Unicode data typed columns right?





This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other

Re: [Catalyst] CSV / UTF-8 / Unicode

[Anthony Lucas]

On 3 July 2013 11:18, Craig Chant  wrote:

>  >> Maybe write a standalone test and take Catalyst and browser quirks
> out of the picture.
>
> I have already done this, I have two SQL wrapper modules one that uses
> DBI::DBD::ODBC and one that uses Win32::ODBC, I applied it to the same
> standalone script that produces CSV output, the only difference between the
> test was one test accessed SQL with the DBI SQL wrapper and one test
> accessed SQL with the Win32::ODBC SQL wrapper, DBI outputted junk chars,
> Win32::ODBC didn't. What else should I be doing to test for the culprit of
> the corruption?
>

You need to see how they are using the ODBC API underneath for handling the
data and encoding.
Setting the trace flag on DBI (i.e. DBI->trace(n)) will expose the
DBD::ODBC activity. I'm not sure of the debugging available for Win32::ODBC.

One thing I would check first is what they are treating the column data as.
If DBD::ODBC is treating the columns as WCHAR but Win32::ODBC is treating
them as CHAR and then doing extra "magic" decoding (or not), well then
you've found a big clue. There has to be different handling or differing
levels of ODBC support somewhere.

I would assume that DBD::ODBC is doing "the right thing", and something
else is amiss upstream (but well, never assume with Unicode handling, so
make sure with the trace).




> >> Also, you are aware that your data will probably be coming back as UCS2
> if you're using SQL Server right?
>
>
> No, what is UCS2 and is this handled differently in DBI::DBD::ODBC vs
> Win32::ODBC ?
>
>
>
>From what I understand, is ultimately what you've got happening?:
Original Input Data -> SQL Client -> Database Driver -> Database (UCS2) ->
Windows ODBC Driver -> DBD::ODBC -> Catalyst(?)

If so, since you're storing the data as Unicode and the database driver
knows this (because your column type is NVARCHAR etc.), conversion to UCS2
happens at the driver stage on Windows. This is lossless between the
different Unicodes, so just make sure your input is actual good Unicode up
to that point and your data is being stored correctly.

Your data is being stored in Unicode data typed columns right?
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Hi all , thanks for the replies.



>> $xls is a scalar you have in memory of the comma separated value data?



Yes, correct, and when output direct to $c->response->body , catalyst server 
falls over with an out of memory error and reboots itself. I had an IRC 
discussion about this and it was acknowledged as a limitation of Catalyst and 
the solution was to wrap it up in a IO:File object.



>> but how Windows decides how to open files is something I'm not familiar with.
>> That depends on the Browser setting per mime type and not the OS.
>> In Firefox you can define it in Preferences / Applications.



Perhaps, but it appears default behaviour in IE / FF / Opera / Chrome to open 
CSV as an XLS spread sheet in Excel seems to require that MIME type and works 
exactly as desired when used.



Reading the RFC link it states no MIME has ever officially been registered of 
CSV but does give the text/csv MIME type, yet as mentioned I didn't get the 
desired behaviour when used, is there an issue with using the MIME type I have, 
if it is yielding desired results?



>> Maybe write a standalone test and take Catalyst and browser quirks out of 
>> the picture.



I have already done this, I have two SQL wrapper modules one that uses 
DBI::DBD::ODBC and one that uses Win32::ODBC, I applied it to the same 
standalone script that produces CSV output, the only difference between the 
test was one test accessed SQL with the DBI SQL wrapper and one test accessed 
SQL with the Win32::ODBC SQL wrapper, DBI outputted junk chars, Win32::ODBC 
didn't. What else should I be doing to test for the culprit of the corruption?



>> Also, you are aware that your data will probably be coming back as UCS2 if 
>> you're using SQL Server right?

No, what is UCS2 and is this handled differently in DBI::DBD::ODBC vs 
Win32::ODBC ?



Thanks,

Craig






From: Anthony Lucas [anthonyjlu...@gmail.com]
Sent: 03 July 2013 06:42
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

I don't see how DBI can be corrupting your data. DBD::ODBC, or more 
specifically the actual driver it is using will be more suspect in that case.

In either case, setting DBI's trace flag to something high should provide some 
insight  who's doing what, and you can make sure they are being handled as wide 
characters. ODBC drivers are renowned for providing incomplete APIs and lying 
about stuff.

Maybe write a standalone test and take Catalyst and browser quirks out of the 
picture.
Also, you are aware that your data will probably be coming back as UCS2 if 
you're using SQL Server right?


On 2 July 2013 17:29, Craig Chant 
mailto:cr...@homeloanpartnership.com>> wrote:

>> All the above seems overkill.   I suspect what you want is closer to this: 
>> (but see notes below).



Tried that, didn’t work, ended up in a long Catalyst discussion where it was 
worked out that I needed to wrap any XLS output to an IO:FILE handle otherwise 
Catalyst dies with an “out of memory” error something to do with streaming data 
support issues in Catalyst so the work round is to wrap the output into an 
IO:File object.



>>Second, be aware that $c->response->content_length(length($xls));



Yes, I was doing the encode then using Length (I did read on perldocs about 
requesting the length against the octet) , either way, the length was the least 
of my worries, keeping Catalyst from falling over with ‘Wide Character’ errors, 
or not getting garbage was my main concern.



And yes, the output is CSV not strictly XLS but I have been told and looked it 
up on the net that 'application/vnd.ms-excel'  Is the correct MIME header to 
pass for CSV that you want MS Excel to open.



Of course, if I have the wrong MIME header for CSV -> MS Excel please can you 
provide the correct one, as it took me a long time to find that one, as the box 
standard ‘text/csv’ does not work properly when opened in MS Excel.



Though as it appears DBI is corrupting my Unicode data, it might be related to 
that rather than CSV->MS Excel per sae!





From: Bill Moseley [mailto:mose...@hank.org<mailto:mose...@hank.org>]
Sent: 02 July 2013 16:59

To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode





On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant 
mailto:cr...@homeloanpartnership.com>> wrote:

# output header
$c->response->content_type('application/vnd.ms-excel');
$c->response->content_length(length($xls));
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');

# create an IO::File for Catalyst
use IO::File;
my $iof = IO::File->new;

$iof->open(\$xls, "r");
$iof->binmode(":encoding(UTF-8)");

  

Re: [Catalyst] CSV / UTF-8 / Unicode

[Anthony Lucas]

I don't see how DBI can be corrupting your data. DBD::ODBC, or more
specifically the actual driver it is using will be more suspect in that
case.

In either case, setting DBI's trace flag to something high should provide
some insight  who's doing what, and you can make sure they are being
handled as wide characters. ODBC drivers are renowned for providing
incomplete APIs and lying about stuff.

Maybe write a standalone test and take Catalyst and browser quirks out of
the picture.
Also, you are aware that your data will probably be coming back as UCS2 if
you're using SQL Server right?


On 2 July 2013 17:29, Craig Chant  wrote:

>  >> All the above seems overkill.   I suspect what you want is closer to
> this: (but see notes below).
>
>
>
> Tried that, didn’t work, ended up in a long Catalyst discussion where it
> was worked out that I needed to wrap any XLS output to an IO:FILE handle
> otherwise Catalyst dies with an “out of memory” error something to do with
> streaming data support issues in Catalyst so the work round is to wrap the
> output into an IO:File object.
>
>
>
> >>Second, be aware that $c->response->content_length(length($xls));
>
>
>
> Yes, I was doing the encode then using Length (I did read on perldocs
> about requesting the length against the octet) , either way, the length was
> the least of my worries, keeping Catalyst from falling over with ‘Wide
> Character’ errors, or not getting garbage was my main concern.
>
>
>
> And yes, the output is CSV not strictly XLS but I have been told and
> looked it up on the net that 'application/vnd.ms-excel'  Is the correct
> MIME header to pass for CSV that you want MS Excel to open.
>
>
>
> Of course, if I have the wrong MIME header for CSV -> MS Excel please can
> you provide the correct one, as it took me a long time to find that one, as
> the box standard ‘text/csv’ does not work properly when opened in MS Excel.
>
>
>
> Though as it appears DBI is corrupting my Unicode data, it might be
> related to that rather than CSV->MS Excel per sae!
>
>
>
>
>
> *From:* Bill Moseley [mailto:mose...@hank.org]
> *Sent:* 02 July 2013 16:59
>
> *To:* The elegant MVC web framework
> *Subject:* Re: [Catalyst] CSV / UTF-8 / Unicode
>
>
>
>
>
> On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant 
> wrote:
>
>
> # output header
> $c->response->content_type('application/vnd.ms-excel');
> $c->response->content_length(length($xls));
> $c->response->header(Content_Disposition =>
> 'attachment;filename=NBCS_Export.csv');
>
>
> # create an IO::File for Catalyst
> use IO::File;
> my $iof = IO::File->new;
>
> $iof->open(\$xls, "r");
> $iof->binmode(":encoding(UTF-8)");
>
> # output XLS data
> $c->response->body($iof);
>
>
>
> All the above seems overkill.   I suspect what you want is closer to this:
> (but see notes below).
>
>
>
> $c->response->content_type('text/csv');
>
> $c->response->body($xls);
> $c->response->header(Content_Disposition =>
> 'attachment;filename=NBCS_Export.csv');
>
>
>
> Then with that content type the plugin would encode $xls as utf8 and add
> ;charset=utf8 (or whatever it is configured to encode as).
>
>
>
> Notes:
>
>
>
> First, you are not returning Excel, so the content type is not what you
> first listed above, right?
>
>
>
> Second, be aware that $c->response->content_length(length($xls)); could be
> very wrong.  If $xls is really CSV text AND it's decoded then length($xls)
> is the length in characters, not octets.   Don't set the content length.
>
>
>
>
>
> Third, Catalyst::Plugin::Unicode::Encoding, IMO, has some issues.
>
>
>
> The plugin limits to just these content types.
>
>
>
> return $c->next::method(@_)
>
>   unless $c->response->content_type =~ /^text|xml$|javascript$/;
>
>
>
> Then it does this:
>
>
>
> $c->response->body( $c->encoding->encode( $body, $CHECK ) )
>
> if ref(\$body) eq 'SCALAR';
>
>
>
> Personally, I think the correct approach is to only encode * character* data
> -- that is check to see if the utf8 flag is set before calling encode.
>
>
>
> Maybe limit to the content types listed above, but throw an exception for
> other content types where the body is a scalar AND has the utf8 flag on.
>  After all, we can only write out octets or else we get the Wide

Re: [Catalyst] CSV / UTF-8 / Unicode

[Alexander Hartmaier]

On 2013-07-02 19:26, Bill Moseley wrote:


On Tue, Jul 2, 2013 at 9:29 AM, Craig Chant 
mailto:cr...@homeloanpartnership.com>> wrote:
>> All the above seems overkill.   I suspect what you want is closer to this: 
>> (but see notes below).

Tried that, didn't work, ended up in a long Catalyst discussion where it was 
worked out that I needed to wrap any XLS output to an IO:FILE handle otherwise 
Catalyst dies with an "out of memory" error something to do with streaming data 
support issues in Catalyst so the work round is to wrap the output into an 
IO:File object.

$xls is a scalar you have in memory of the comma separated value data?



>>Second, be aware that $c->response->content_length(length($xls));

Yes, I was doing the encode then using Length (I did read on perldocs about 
requesting the length against the octet) , either way, the length was the least 
of my worries, keeping Catalyst from falling over with 'Wide Character' errors, 
or not getting garbage was my main concern.

And yes, the output is CSV not strictly XLS but I have been told and looked it 
up on the net that 'application/vnd.ms-excel'  Is the correct MIME header to 
pass for CSV that you want MS Excel to open.

http://www.ietf.org/rfc/rfc4180.txt, but how Windows decides how to open files 
is something I'm not familiar with.
That depends on the Browser setting per mime type and not the OS.
In Firefox you can define it in Preferences / Applications.



--
Bill Moseley
mose...@hank.org


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/




*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
Handelsgericht Wien, FN 79340b
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
Notice: This e-mail contains information that is confidential and may be 
privileged.
If you are not the intended recipient, please notify the sender and then
delete this e-mail immediately.
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Bill Moseley]

On Tue, Jul 2, 2013 at 9:29 AM, Craig Chant
wrote:

>  >> All the above seems overkill.   I suspect what you want is closer to
> this: (but see notes below).
>
>
>
> Tried that, didn’t work, ended up in a long Catalyst discussion where it
> was worked out that I needed to wrap any XLS output to an IO:FILE handle
> otherwise Catalyst dies with an “out of memory” error something to do with
> streaming data support issues in Catalyst so the work round is to wrap the
> output into an IO:File object.
>

$xls is a scalar you have in memory of the comma separated value data?



>
>
> >>Second, be aware that $c->response->content_length(length($xls));
>
>
>
> Yes, I was doing the encode then using Length (I did read on perldocs
> about requesting the length against the octet) , either way, the length was
> the least of my worries, keeping Catalyst from falling over with ‘Wide
> Character’ errors, or not getting garbage was my main concern.
>
>
>
> And yes, the output is CSV not strictly XLS but I have been told and
> looked it up on the net that 'application/vnd.ms-excel'  Is the correct
> MIME header to pass for CSV that you want MS Excel to open.
>

http://www.ietf.org/rfc/rfc4180.txt, but how Windows decides how to open
files is something I'm not familiar with.



-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

>> All the above seems overkill.   I suspect what you want is closer to this: 
>> (but see notes below).

Tried that, didn't work, ended up in a long Catalyst discussion where it was 
worked out that I needed to wrap any XLS output to an IO:FILE handle otherwise 
Catalyst dies with an "out of memory" error something to do with streaming data 
support issues in Catalyst so the work round is to wrap the output into an 
IO:File object.

>>Second, be aware that $c->response->content_length(length($xls));

Yes, I was doing the encode then using Length (I did read on perldocs about 
requesting the length against the octet) , either way, the length was the least 
of my worries, keeping Catalyst from falling over with 'Wide Character' errors, 
or not getting garbage was my main concern.

And yes, the output is CSV not strictly XLS but I have been told and looked it 
up on the net that 'application/vnd.ms-excel'  Is the correct MIME header to 
pass for CSV that you want MS Excel to open.

Of course, if I have the wrong MIME header for CSV -> MS Excel please can you 
provide the correct one, as it took me a long time to find that one, as the box 
standard 'text/csv' does not work properly when opened in MS Excel.

Though as it appears DBI is corrupting my Unicode data, it might be related to 
that rather than CSV->MS Excel per sae!


From: Bill Moseley [mailto:mose...@hank.org]
Sent: 02 July 2013 16:59
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode


On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant 
mailto:cr...@homeloanpartnership.com>> wrote:

# output header
$c->response->content_type('application/vnd.ms-excel');
$c->response->content_length(length($xls));
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');

# create an IO::File for Catalyst
use IO::File;
my $iof = IO::File->new;

$iof->open(\$xls, "r");
$iof->binmode(":encoding(UTF-8)");

# output XLS data
$c->response->body($iof);

All the above seems overkill.   I suspect what you want is closer to this: (but 
see notes below).

$c->response->content_type('text/csv');
$c->response->body($xls);
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');

Then with that content type the plugin would encode $xls as utf8 and add 
;charset=utf8 (or whatever it is configured to encode as).

Notes:

First, you are not returning Excel, so the content type is not what you first 
listed above, right?

Second, be aware that $c->response->content_length(length($xls)); could be very 
wrong.  If $xls is really CSV text AND it's decoded then length($xls) is the 
length in characters, not octets.   Don't set the content length.


Third, Catalyst::Plugin::Unicode::Encoding, IMO, has some issues.

The plugin limits to just these content types.

return $c->next::method(@_)
  unless $c->response->content_type =~ /^text|xml$|javascript$/;

Then it does this:

$c->response->body( $c->encoding->encode( $body, $CHECK ) )
if ref(\$body) eq 'SCALAR';

Personally, I think the correct approach is to only encode character data -- 
that is check to see if the utf8 flag is set before calling encode.

Maybe limit to the content types listed above, but throw an exception for other 
content types where the body is a scalar AND has the utf8 flag on.  After all, 
we can only write out octets or else we get the Wide Character error.






--
Bill Moseley
mose...@hank.org<mailto:mose...@hank.org>
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communications 
through its network. No representative or employee of HomeLoan Partnership has 
the authority to enter into any contract on behalf of HomeLoan Partnership by 
email. HomeLoan Partnership is a 

Re: [Catalyst] CSV / UTF-8 / Unicode

[Bill Moseley]

Before we get a long utf8 discussion here:

On Tue, Jul 2, 2013 at 8:58 AM, Bill Moseley  wrote:

>
> Personally, I think the correct approach is to only encode *character* data
> -- that is check to see if the utf8 flag is set before calling encode.
>

I say that with the caveat that ALL textual input data is correctly decoded
-- as it should be.  That is, everything that was decoded into characters
has the utf8 flag set.

$ perl -MEncode -le 'print "yes, this is characters" if Encode::is_utf8(
Encode::decode( "ASCII", "This is ASCII"))'
yes, this is characters



-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Bill Moseley]

On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant
wrote:

>
> # output header
> $c->response->content_type('application/vnd.ms-excel');
> $c->response->content_length(length($xls));
> $c->response->header(Content_Disposition =>
> 'attachment;filename=NBCS_Export.csv');
>
> # create an IO::File for Catalyst
> use IO::File;
> my $iof = IO::File->new;
>
> $iof->open(\$xls, "r");
> $iof->binmode(":encoding(UTF-8)");
>
> # output XLS data
> $c->response->body($iof);
>

All the above seems overkill.   I suspect what you want is closer to this:
(but see notes below).

$c->response->content_type('text/csv');
$c->response->body($xls);
$c->response->header(Content_Disposition =>
'attachment;filename=NBCS_Export.csv');

Then with that content type the plugin would encode $xls as utf8 and add
;charset=utf8 (or whatever it is configured to encode as).

Notes:

First, you are not returning Excel, so the content type is not what you
first listed above, right?

Second, be aware that $c->response->content_length(length($xls)); could be
very wrong.  If $xls is really CSV text AND it's decoded then length($xls)
is the length in characters, not octets.   Don't set the content length.


Third, Catalyst::Plugin::Unicode::Encoding, IMO, has some issues.

The plugin limits to just these content types.

return $c->next::method(@_)
  unless $c->response->content_type =~ /^text|xml$|javascript$/;

Then it does this:

$c->response->body( $c->encoding->encode( $body, $CHECK ) )
if ref(\$body) eq 'SCALAR';

Personally, I think the correct approach is to only encode *character* data
-- that is check to see if the utf8 flag is set before calling encode.

Maybe limit to the content types listed above, but throw an exception for
other content types where the body is a scalar AND has the utf8 flag on.
 After all, we can only write out octets or else we get the Wide Character
error.






-- 
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

I looked at that link but it doesn't have anything on DBD::ODBC that I could 
see.

Interesting enough utf8::is_utf8($xls) returns true, but it isn't because the 
output is garbage?

Also as utf8::is_utf8 claims the string is UTF-8,  why does decode baulk with 
'cannot decode string with wide characters'?


-Original Message-
From: Mike Whitaker [mailto:m...@altrion.org]
Sent: 02 July 2013 14:58
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

On 2 Jul 2013, at 14:36, Mike Whitaker  wrote:
> On 2 Jul 2013, at 14:34, Craig Chant  wrote:
>>
>> So either I refactor my SQL model wrappers under Catalyst to use 
>> Win32::ODBC, or I find out why DBI is corrupting my SQL data.
>
> Be a hero. Do the latter :D :D

..and have a read of http://juerd.nl/site.plp/perluniadvice

You may also find that calling utf8::is_utf8() on the strings that come back 
from your DB may be revealing (this is one case in which you may use this 
without people screaming N!), as may 
http://search.cpan.org/~ilyaz/Devel-Peek-0.96/Peek.pm for peeking into the 
innnards of Perl strings.


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communications 
through its network. No representative or employee of HomeLoan Partnership has 
the authority to enter into any contract on behalf of HomeLoan Partnership by 
email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
registered in England and Wales with Registration Number 5011722. Registered 
office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is 
authorised and regulated by the Financial Conduct Authority.

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

On 2 Jul 2013, at 14:36, Mike Whitaker  wrote:
> On 2 Jul 2013, at 14:34, Craig Chant  wrote:
>> 
>> So either I refactor my SQL model wrappers under Catalyst to use 
>> Win32::ODBC, or I find out why DBI is corrupting my SQL data.
> 
> Be a hero. Do the latter :D :D

..and have a read of http://juerd.nl/site.plp/perluniadvice 

You may also find that calling utf8::is_utf8() on the strings that come back 
from your DB may be revealing (this is one case in which you may use this 
without people screaming N!), as may 
http://search.cpan.org/~ilyaz/Devel-Peek-0.96/Peek.pm for peeking into the 
innnards of Perl strings.


___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

On 2 Jul 2013, at 14:34, Craig Chant  wrote:
> 
> So either I refactor my SQL model wrappers under Catalyst to use Win32::ODBC, 
> or I find out why DBI is corrupting my SQL data.

Be a hero. Do the latter :D :D
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Hi Alexander,

Well I think I found the problem... the DBI module!

I went back to my original SQL wrapper for Win32::ODBC in my original 
procedural script application before porting to Catalyst (where I moved to DBI).

All my output to XLS in the old script using Win32::ODBC works perfectly, no 
funny chars in sight!

So it has to be flaky Windows support in the DBI module to blame wouldn't you 
say?

So either I refactor my SQL model wrappers under Catalyst to use Win32::ODBC, 
or I find out why DBI is corrupting my SQL data.

No wonder I was going round in circles trying to UTF-8 my data!

DBI on Windows has always been a thorn in my side especially with the "long 
read" issues that occur!

I guess I need to head to the Perl forums for DBI support, though it might be 
nice if MST chimed in as I know he's an expert on all things DBI/DBIC.

Is there a switch or flag or something I am missing to make DBI work properly 
under DBD::ODBC?

Oh and I did a test with DBI in my old procedural application script and got 
the same funny characters as I did with DBI under Catalyst, so I think we can 
safely say it is DBI with the problem and not Catalyst.

Any input greatly received.

Regards,
1DMF


-Original Message-
From: Alexander Hartmaier [mailto:alexander.hartma...@t-systems.at]
Sent: 02 July 2013 13:49
To: catalyst@lists.scsys.co.uk
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

On 2013-07-02 14:20, Craig Chant wrote:
> Hi Alexander,
>
> I have read the DBD::ODBC docs and it seems Unicode is ON by default
>
> " Enabling and Disabling Unicode support
>
> On Windows Unicode support is enabled by default"
>
> So do I assume the data coming from the DB is Unicode encoded already?
>
> if so , why does decode fall over?
>
> I don't seem able to decode or encode without the app crashing?
>
> Do I have non-Unicode chars perhaps?
>
> Craig.
I'd suggest to test it outside of Catalyst with a simple standalone script.
I have neither experience with Perl on Windows nor with ODBC and MSSQL.
Ask on the DBIC list or on IRC, some guys use MSSQL successfully there.

BR Alex

>
> -Original Message-
> From: Alexander Hartmaier [mailto:alexander.hartma...@t-systems.at]
> Sent: 02 July 2013 12:53
> To: catalyst@lists.scsys.co.uk
> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>
> On 2013-07-02 13:45, Craig Chant wrote:
>> Hi Mike,
>>
>> The data is coming from MS SQL 2008 R2, I have checked the DB and the column 
>> is defined as data type nvarchar(max) , so is set to Unicode and when 
>> viewing data via SQL Studio Manager, the Unicode characters display fine.
>>
>> I am accessing the DB via DBI and have set on the connection string..
>> {'RaiseError' => 1, 'mysql_enable_utf8' => 1}
> MSSQL != MySQL
>> Though I'm not sure if the utf8 setting has any affect when using an MS ODBC 
>> driver as the flag implies it's for mysql ?
> No it doesn't, read your DBD's docs.
>> I tried to apply  my $chars = decode('UTF-8', $xls); and I got an error 
>> saying you cannot decode a string with wide characters.
>>
>> I don't know what the string is, but when you try to decode you get a wide 
>> character error and when you try to encode you get a wide character error.
>>
>> If I don't do anything to it and just output it, no error, but the data is 
>> full of junk chars.
>>
>> Where do I go from here?
> Ensure that the data coming out from your database is properly decoded, then 
> make sure it's properly encoded when outputting it.
> Never mess with Perls' internal representation of the data by dis-/enabling 
> the utf-8 bit.
>> Thanks,
>> Craig.
>>
>>
>>
>> -Original Message-
>> From: Mike Whitaker [mailto:m...@altrion.org]
>> Sent: 02 July 2013 12:27
>> To: The elegant MVC web framework
>> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>>
>>
>>>   # encode UTF8 octet
>>>use Encode qw(decode encode);
>>>my $octets = encode('UTF-8', $xls);
>> OUt of curiousity, where's the data in $xls come from, and what do you know 
>> about its encoding and bytes vs characters status?
>> ___
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/ This Email and any
>> attachments contain confidential information and is intended solely for the 
>> individual to whom it is addressed. If this Email has

Re: [Catalyst] CSV / UTF-8 / Unicode

[Alexander Hartmaier]

On 2013-07-02 14:20, Craig Chant wrote:
> Hi Alexander,
>
> I have read the DBD::ODBC docs and it seems Unicode is ON by default
>
> " Enabling and Disabling Unicode support
>
> On Windows Unicode support is enabled by default"
>
> So do I assume the data coming from the DB is Unicode encoded already?
>
> if so , why does decode fall over?
>
> I don't seem able to decode or encode without the app crashing?
>
> Do I have non-Unicode chars perhaps?
>
> Craig.
I'd suggest to test it outside of Catalyst with a simple standalone script.
I have neither experience with Perl on Windows nor with ODBC and MSSQL.
Ask on the DBIC list or on IRC, some guys use MSSQL successfully there.

BR Alex

>
> -Original Message-
> From: Alexander Hartmaier [mailto:alexander.hartma...@t-systems.at]
> Sent: 02 July 2013 12:53
> To: catalyst@lists.scsys.co.uk
> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>
> On 2013-07-02 13:45, Craig Chant wrote:
>> Hi Mike,
>>
>> The data is coming from MS SQL 2008 R2, I have checked the DB and the column 
>> is defined as data type nvarchar(max) , so is set to Unicode and when 
>> viewing data via SQL Studio Manager, the Unicode characters display fine.
>>
>> I am accessing the DB via DBI and have set on the connection string..
>> {'RaiseError' => 1, 'mysql_enable_utf8' => 1}
> MSSQL != MySQL
>> Though I'm not sure if the utf8 setting has any affect when using an MS ODBC 
>> driver as the flag implies it's for mysql ?
> No it doesn't, read your DBD's docs.
>> I tried to apply  my $chars = decode('UTF-8', $xls); and I got an error 
>> saying you cannot decode a string with wide characters.
>>
>> I don't know what the string is, but when you try to decode you get a wide 
>> character error and when you try to encode you get a wide character error.
>>
>> If I don't do anything to it and just output it, no error, but the data is 
>> full of junk chars.
>>
>> Where do I go from here?
> Ensure that the data coming out from your database is properly decoded, then 
> make sure it's properly encoded when outputting it.
> Never mess with Perls' internal representation of the data by dis-/enabling 
> the utf-8 bit.
>> Thanks,
>> Craig.
>>
>>
>>
>> -Original Message-
>> From: Mike Whitaker [mailto:m...@altrion.org]
>> Sent: 02 July 2013 12:27
>> To: The elegant MVC web framework
>> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>>
>>
>>>   # encode UTF8 octet
>>>use Encode qw(decode encode);
>>>my $octets = encode('UTF-8', $xls);
>> OUt of curiousity, where's the data in $xls come from, and what do you know 
>> about its encoding and bytes vs characters status?
>> ___
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/ This Email and any attachments
>> contain confidential information and is intended solely for the individual 
>> to whom it is addressed. If this Email has been misdirected, please notify 
>> the author as soon as possible. If you are not the intended recipient you 
>> must not disclose, distribute, copy, print or rely on any of the information 
>> contained, and all copies must be deleted immediately. Whilst we take 
>> reasonable steps to try to identify any software viruses, any attachments to 
>> this e-mail may nevertheless contain viruses, which our anti-virus software 
>> has failed to identify. You should therefore carry out your own anti-virus 
>> checks before opening any documents. HomeLoan Partnership will not accept 
>> any liability for damage caused by computer viruses emanating from any 
>> attachment or other document supplied with this e-mail. HomeLoan Partnership 
>> reserves the right to monitor and archive all e-mail communications through 
>> its network. No representative or employee of HomeLoan Partnership has the 
>> authority to enter into any contract on behalf of HomeLoan Partnership by 
>> email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
>> registered in England and Wales with Registration Number 5011722. Registered 
>> office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is 
>> authorised and regulated by the Financial Conduct Authority.
>>
>> 

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Hi Alexander,

I have read the DBD::ODBC docs and it seems Unicode is ON by default

" Enabling and Disabling Unicode support

On Windows Unicode support is enabled by default"

So do I assume the data coming from the DB is Unicode encoded already?

if so , why does decode fall over?

I don't seem able to decode or encode without the app crashing?

Do I have non-Unicode chars perhaps?

Craig.

-Original Message-
From: Alexander Hartmaier [mailto:alexander.hartma...@t-systems.at]
Sent: 02 July 2013 12:53
To: catalyst@lists.scsys.co.uk
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

On 2013-07-02 13:45, Craig Chant wrote:
> Hi Mike,
>
> The data is coming from MS SQL 2008 R2, I have checked the DB and the column 
> is defined as data type nvarchar(max) , so is set to Unicode and when viewing 
> data via SQL Studio Manager, the Unicode characters display fine.
>
> I am accessing the DB via DBI and have set on the connection string..
> {'RaiseError' => 1, 'mysql_enable_utf8' => 1}
MSSQL != MySQL
>
> Though I'm not sure if the utf8 setting has any affect when using an MS ODBC 
> driver as the flag implies it's for mysql ?
No it doesn't, read your DBD's docs.
>
> I tried to apply  my $chars = decode('UTF-8', $xls); and I got an error 
> saying you cannot decode a string with wide characters.
>
> I don't know what the string is, but when you try to decode you get a wide 
> character error and when you try to encode you get a wide character error.
>
> If I don't do anything to it and just output it, no error, but the data is 
> full of junk chars.
>
> Where do I go from here?
Ensure that the data coming out from your database is properly decoded, then 
make sure it's properly encoded when outputting it.
Never mess with Perls' internal representation of the data by dis-/enabling the 
utf-8 bit.
>
> Thanks,
> Craig.
>
>
>
> -Original Message-
> From: Mike Whitaker [mailto:m...@altrion.org]
> Sent: 02 July 2013 12:27
> To: The elegant MVC web framework
> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>
>
>>   # encode UTF8 octet
>>use Encode qw(decode encode);
>>my $octets = encode('UTF-8', $xls);
> OUt of curiousity, where's the data in $xls come from, and what do you know 
> about its encoding and bytes vs characters status?
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/ This Email and any attachments
> contain confidential information and is intended solely for the individual to 
> whom it is addressed. If this Email has been misdirected, please notify the 
> author as soon as possible. If you are not the intended recipient you must 
> not disclose, distribute, copy, print or rely on any of the information 
> contained, and all copies must be deleted immediately. Whilst we take 
> reasonable steps to try to identify any software viruses, any attachments to 
> this e-mail may nevertheless contain viruses, which our anti-virus software 
> has failed to identify. You should therefore carry out your own anti-virus 
> checks before opening any documents. HomeLoan Partnership will not accept any 
> liability for damage caused by computer viruses emanating from any attachment 
> or other document supplied with this e-mail. HomeLoan Partnership reserves 
> the right to monitor and archive all e-mail communications through its 
> network. No representative or employee of HomeLoan Partnership has the 
> authority to enter into any contract on behalf of HomeLoan Partnership by 
> email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
> registered in England and Wales with Registration Number 5011722. Registered 
> office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is 
> authorised and regulated by the Financial Conduct Authority.
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/




*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien Handelsgericht Wien, FN 79340b
*"*"*"*"*"*"*"*"

Re: [Catalyst] CSV / UTF-8 / Unicode

[Alexander Hartmaier]

On 2013-07-02 13:45, Craig Chant wrote:
> Hi Mike,
>
> The data is coming from MS SQL 2008 R2, I have checked the DB and the column 
> is defined as data type nvarchar(max) , so is set to Unicode and when viewing 
> data via SQL Studio Manager, the Unicode characters display fine.
>
> I am accessing the DB via DBI and have set on the connection string.. 
> {'RaiseError' => 1, 'mysql_enable_utf8' => 1}
MSSQL != MySQL
>
> Though I'm not sure if the utf8 setting has any affect when using an MS ODBC 
> driver as the flag implies it's for mysql ?
No it doesn't, read your DBD's docs.
>
> I tried to apply  my $chars = decode('UTF-8', $xls); and I got an error 
> saying you cannot decode a string with wide characters.
>
> I don't know what the string is, but when you try to decode you get a wide 
> character error and when you try to encode you get a wide character error.
>
> If I don't do anything to it and just output it, no error, but the data is 
> full of junk chars.
>
> Where do I go from here?
Ensure that the data coming out from your database is properly decoded,
then make sure it's properly encoded when outputting it.
Never mess with Perls' internal representation of the data by
dis-/enabling the utf-8 bit.
>
> Thanks,
> Craig.
>
>
>
> -----Original Message-
> From: Mike Whitaker [mailto:m...@altrion.org]
> Sent: 02 July 2013 12:27
> To: The elegant MVC web framework
> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode
>
>
>>   # encode UTF8 octet
>>use Encode qw(decode encode);
>>my $octets = encode('UTF-8', $xls);
> OUt of curiousity, where's the data in $xls come from, and what do you know 
> about its encoding and bytes vs characters status?
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
> This Email and any attachments contain confidential information and is 
> intended solely for the individual to whom it is addressed. If this Email has 
> been misdirected, please notify the author as soon as possible. If you are 
> not the intended recipient you must not disclose, distribute, copy, print or 
> rely on any of the information contained, and all copies must be deleted 
> immediately. Whilst we take reasonable steps to try to identify any software 
> viruses, any attachments to this e-mail may nevertheless contain viruses, 
> which our anti-virus software has failed to identify. You should therefore 
> carry out your own anti-virus checks before opening any documents. HomeLoan 
> Partnership will not accept any liability for damage caused by computer 
> viruses emanating from any attachment or other document supplied with this 
> e-mail. HomeLoan Partnership reserves the right to monitor and archive all 
> e-mail communications through its network. No representative or employee of 
> HomeLoan Partnership has the authority to enter into any contract on behalf 
> of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H 
> L Partnership Limited, registered in England and Wales with Registration 
> Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L 
> Partnership Limited is authorised and regulated by the Financial Conduct 
> Authority.
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/




*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien
Handelsgericht Wien, FN 79340b
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*
Notice: This e-mail contains information that is confidential and may be 
privileged.
If you are not the intended recipient, please notify the sender and then
delete this e-mail immediately.
*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Hi Mike,

The data is coming from MS SQL 2008 R2, I have checked the DB and the column is 
defined as data type nvarchar(max) , so is set to Unicode and when viewing data 
via SQL Studio Manager, the Unicode characters display fine.

I am accessing the DB via DBI and have set on the connection string.. 
{'RaiseError' => 1, 'mysql_enable_utf8' => 1}

Though I'm not sure if the utf8 setting has any affect when using an MS ODBC 
driver as the flag implies it's for mysql ?

I tried to apply  my $chars = decode('UTF-8', $xls); and I got an error saying 
you cannot decode a string with wide characters.

I don't know what the string is, but when you try to decode you get a wide 
character error and when you try to encode you get a wide character error.

If I don't do anything to it and just output it, no error, but the data is full 
of junk chars.

Where do I go from here?

Thanks,
Craig.



-Original Message-
From: Mike Whitaker [mailto:m...@altrion.org]
Sent: 02 July 2013 12:27
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode


>   # encode UTF8 octet
>use Encode qw(decode encode);
>my $octets = encode('UTF-8', $xls);

OUt of curiousity, where's the data in $xls come from, and what do you know 
about its encoding and bytes vs characters status?
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communications 
through its network. No representative or employee of HomeLoan Partnership has 
the authority to enter into any contract on behalf of HomeLoan Partnership by 
email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
registered in England and Wales with Registration Number 5011722. Registered 
office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is 
authorised and regulated by the Financial Conduct Authority.

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

>   # encode UTF8 octet
>use Encode qw(decode encode);
>my $octets = encode('UTF-8', $xls);

OUt of curiousity, where's the data in $xls come from, and what do you know 
about its encoding and bytes vs characters status?
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

well Lars said that binmode(":encoding...") wasn't encoding.

I thought it was, I very confused here!

Anyway, I've also tried...



# encode UTF8 octet
use Encode qw(decode encode);
my $octets = encode('UTF-8', $xls);

# output header
$c->response->content_type('application/vnd.ms-excel');
$c->response->content_length(length($octets));
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');

# create an IO::File for Catalyst
use IO::File;
my $iof = IO::File->new;

$iof->open(\$octets, "r");
#$iof->binmode(":encoding(UTF-8)");

# output XLS data
$c->response->body($iof);

# close file
undef $iof;

---

But still if I use the binmode encoding line catalyst falls over with 'wide 
character' and if I don't I get a spread sheet full of junk characters?

Either way, it's not working, why would this be?

Craig.

-Original Message-
From: Mike Whitaker [mailto:m...@altrion.org]
Sent: 02 July 2013 11:47
To: The elegant MVC web framework
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

On 2 Jul 2013, at 10:59, Craig Chant  wrote:
>
>utf8::encode($xls);
>
>$iof->binmode(":encoding(UTF-8)");

...erm, haven't you now encoded twice?

Also? Don't use utf8::encode - if you can't use IO layers etc, use the general 
Encode module.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communications 
through its network. No representative or employee of HomeLoan Partnership has 
the authority to enter into any contract on behalf of HomeLoan Partnership by 
email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
registered in England and Wales with Registration Number 5011722. Registered 
office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is 
authorised and regulated by the Financial Conduct Authority.

___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

On 2 Jul 2013, at 10:59, Craig Chant  wrote:
> 
>utf8::encode($xls);
> 
>$iof->binmode(":encoding(UTF-8)");

...erm, haven't you now encoded twice?

Also? Don't use utf8::encode - if you can't use IO layers etc, use the general 
Encode module.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Mike Whitaker]

On 2 Jul 2013, at 10:38, Craig Chant  wrote:
> 
> I thought all strings in perl were Unicode ? or at least that's what the 
> Unicode::Encoding plugin is for isnt' it?

No.

Decoded strings in Perl are in a character representation which is magic and 
you don't need to know about. Assuming that representation is Unicode will bite 
you on the backside when you most want it not to.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

To follow up having looked at perldoc http://perldoc.perl.org/utf8.html 

I have just tried...
-

# encode UTF8
use utf8;
utf8::encode($xls);

# output header
$c->response->content_type('application/vnd.ms-excel');
$c->response->content_length(length($xls));
$c->response->header(Content_Disposition => 
'attachment;filename=NBCS_Export.csv');

# create an IO::File for Catalyst
use IO::File;
my $iof = IO::File->new;

$iof->open(\$xls, "r");
$iof->binmode(":encoding(UTF-8)");

# output XLS data
$c->response->body($iof);

# close file
undef $iof;

---

And still get exactly the same error...

[error] Caught exception in engine "Wide character in syswrite at 
C:/Perl/lib/IO/Handle.pm line 474."

Do I need to use something else to encode to utf8?

Thanks,
Craig.

-Original Message-
From: Craig Chant [mailto:cr...@homeloanpartnership.com]
Sent: 02 July 2013 10:38
To: The elegant MVC web framework
Subject: RE: [Catalyst] CSV / UTF-8 / Unicode

Hi Lars,

Thanks for the reply

>> The problem is rather that Catalyst::Response::body wants octets

Sorry, you've lost me, I thought response body wanted an IO::File Handle which 
is what I have given it?

I was also under the impression that $c->response->body( 'Page not found' ); 
was acceptable and that is also a char string not octet?

I seem to be missing something here, please can you elaborate.

>> encode the content again for output as UTF-8

Isn't that what $iof->binmode(":encoding(UTF-8)");  does?

I thought all strings in perl were Unicode ? or at least that's what the 
Unicode::Encoding plugin is for isnt' it?

I have in my app 

use Catalyst qw/
Unicode::Encoding
/;

What's the purpose of this Catgalyst plugin if it isn't making my app Unicode?

How do I encode a string to unicode if it isn't already?

> # close file
> undef $iof;
>> That's not necessary. When the reference falls out of scope, the destructor 
>> that closes the file is called automatically.

Does it do any harm? Is there an issue with cleaning up your own object 
references?

I thought it was good practice to clean up your own references , even if auto 
garbage collection *might* do it for you? It's a habit I have from VBA / 
VB.NET, am I doing something bad?

>> PS: IO::File does not throw exceptions when the file could not be read.
>> Instead I recommend to use the `open` built-in in conjuction

Please note there is no file to be read , this is utilising Perl 5+'s ability 
to use a string as a file handle.

There will always be a string, even if it is empty so the exported XLS will be 
a blank spread sheet.

I think all I need to do is understand how I encode the string before giving it 
to IO::File

Your help is appreciated.

Regards,
Craig.



-Original Message-
From: Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 [mailto:da...@cpan.org]
Sent: 02 July 2013 10:06
To: catalyst@lists.scsys.co.uk
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

> I seem to be having major problems getting the application devel
> server (my_app_server.pl) to accept an IO::File Handle as CSV and
> correctly encode as UTF-8.
No, the code does exactly what you said. The problem is rather that 
Catalyst::Response::body wants octets, not characters. Either pass the handle 
through in raw mode, or encode the content again for output as UTF-8. If the 
input file is not under your control, I recommend going through a manual 
decoding/encoding step instead of IO layers, and make decoding errors fatal 
(see FB_CROAK in <http://p3rl.org/Encode>) in order to catch malformed encoding 
before it's delivered to the client.
One can't be too careful.

> # close file
> undef $iof;
That's not necessary. When the reference falls out of scope, the destructor 
that closes the file is called automatically.

PS: IO::File does not throw exceptions when the file could not be read.
Instead I recommend to use the `open` built-in in conjuction with 
<http://p3rl.org/autodie> for better error checking.
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You 

RE: [Catalyst] CSV / UTF-8 / Unicode

[Craig Chant]

Hi Lars,

Thanks for the reply

>> The problem is rather that Catalyst::Response::body wants octets

Sorry, you've lost me, I thought response body wanted an IO::File Handle which 
is what I have given it?

I was also under the impression that $c->response->body( 'Page not found' ); 
was acceptable and that is also a char string not octet?

I seem to be missing something here, please can you elaborate.

>> encode the content again for output as UTF-8

Isn't that what $iof->binmode(":encoding(UTF-8)");  does?

I thought all strings in perl were Unicode ? or at least that's what the 
Unicode::Encoding plugin is for isnt' it?

I have in my app 

use Catalyst qw/
Unicode::Encoding
/;

What's the purpose of this Catgalyst plugin if it isn't making my app Unicode?

How do I encode a string to unicode if it isn't already?

> # close file
> undef $iof;
>> That's not necessary. When the reference falls out of scope, the destructor 
>> that closes the file is called automatically.

Does it do any harm? Is there an issue with cleaning up your own object 
references?

I thought it was good practice to clean up your own references , even if auto 
garbage collection *might* do it for you? It's a habit I have from VBA / 
VB.NET, am I doing something bad?

>> PS: IO::File does not throw exceptions when the file could not be read.
>> Instead I recommend to use the `open` built-in in conjuction

Please note there is no file to be read , this is utilising Perl 5+'s ability 
to use a string as a file handle.

There will always be a string, even if it is empty so the exported XLS will be 
a blank spread sheet.

I think all I need to do is understand how I encode the string before giving it 
to IO::File

Your help is appreciated.

Regards,
Craig.



-Original Message-----
From: Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯 [mailto:da...@cpan.org]
Sent: 02 July 2013 10:06
To: catalyst@lists.scsys.co.uk
Subject: Re: [Catalyst] CSV / UTF-8 / Unicode

> I seem to be having major problems getting the application devel
> server (my_app_server.pl) to accept an IO::File Handle as CSV and
> correctly encode as UTF-8.
No, the code does exactly what you said. The problem is rather that 
Catalyst::Response::body wants octets, not characters. Either pass the handle 
through in raw mode, or encode the content again for output as UTF-8. If the 
input file is not under your control, I recommend going through a manual 
decoding/encoding step instead of IO layers, and make decoding errors fatal 
(see FB_CROAK in <http://p3rl.org/Encode>) in order to catch malformed encoding 
before it's delivered to the client.
One can't be too careful.

> # close file
> undef $iof;
That's not necessary. When the reference falls out of scope, the destructor 
that closes the file is called automatically.

PS: IO::File does not throw exceptions when the file could not be read.
Instead I recommend to use the `open` built-in in conjuction with 
<http://p3rl.org/autodie> for better error checking.
This Email and any attachments contain confidential information and is intended 
solely for the individual to whom it is addressed. If this Email has been 
misdirected, please notify the author as soon as possible. If you are not the 
intended recipient you must not disclose, distribute, copy, print or rely on 
any of the information contained, and all copies must be deleted immediately. 
Whilst we take reasonable steps to try to identify any software viruses, any 
attachments to this e-mail may nevertheless contain viruses, which our 
anti-virus software has failed to identify. You should therefore carry out your 
own anti-virus checks before opening any documents. HomeLoan Partnership will 
not accept any liability for damage caused by computer viruses emanating from 
any attachment or other document supplied with this e-mail. HomeLoan 
Partnership reserves the right to monitor and archive all e-mail communications 
through its network. No representative or employee of HomeLoan Partnership has 
the authority to enter into any contract on behalf of HomeLoan Partnership by 
email. HomeLoan Partnership is a trading name of H L Partnership Limited, 
registered in England and Wales with Registration Number 5011722. Registered 
office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is 
authorised and regulated by the Financial Conduct Authority.
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: [Catalyst] CSV / UTF-8 / Unicode

[Lars Dɪᴇᴄᴋᴏᴡ 迪拉斯]

> I seem to be having major problems getting the application devel
> server (my_app_server.pl) to accept an IO::File Handle as CSV and
> correctly encode as UTF-8.
No, the code does exactly what you said. The problem is rather that
Catalyst::Response::body wants octets, not characters. Either pass the
handle through in raw mode, or encode the content again for output as
UTF-8. If the input file is not under your control, I recommend going
through a manual decoding/encoding step instead of IO layers, and make
decoding errors fatal (see FB_CROAK in ) in
order to catch malformed encoding before it's delivered to the client.
One can't be too careful.

> # close file
> undef $iof;
That's not necessary. When the reference falls out of scope, the
destructor that closes the file is called automatically.

PS: IO::File does not throw exceptions when the file could not be read.
Instead I recommend to use the `open` built-in in conjuction with
 for better error checking.


signature.asc
Description: PGP signature
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/