Hi Alexander, Well I think I found the problem... the DBI module!
I went back to my original SQL wrapper for Win32::ODBC in my original procedural script application before porting to Catalyst (where I moved to DBI). All my output to XLS in the old script using Win32::ODBC works perfectly, no funny chars in sight! So it has to be flaky Windows support in the DBI module to blame wouldn't you say? So either I refactor my SQL model wrappers under Catalyst to use Win32::ODBC, or I find out why DBI is corrupting my SQL data. No wonder I was going round in circles trying to UTF-8 my data! DBI on Windows has always been a thorn in my side especially with the "long read" issues that occur! I guess I need to head to the Perl forums for DBI support, though it might be nice if MST chimed in as I know he's an expert on all things DBI/DBIC. Is there a switch or flag or something I am missing to make DBI work properly under DBD::ODBC? Oh and I did a test with DBI in my old procedural application script and got the same funny characters as I did with DBI under Catalyst, so I think we can safely say it is DBI with the problem and not Catalyst. Any input greatly received. Regards, 1DMF -----Original Message----- From: Alexander Hartmaier [mailto:alexander.hartma...@t-systems.at] Sent: 02 July 2013 13:49 To: catalyst@lists.scsys.co.uk Subject: Re: [Catalyst] CSV / UTF-8 / Unicode On 2013-07-02 14:20, Craig Chant wrote: > Hi Alexander, > > I have read the DBD::ODBC docs and it seems Unicode is ON by default > > " Enabling and Disabling Unicode support > > On Windows Unicode support is enabled by default" > > So do I assume the data coming from the DB is Unicode encoded already? > > if so , why does decode fall over? > > I don't seem able to decode or encode without the app crashing? > > Do I have non-Unicode chars perhaps? > > Craig. I'd suggest to test it outside of Catalyst with a simple standalone script. I have neither experience with Perl on Windows nor with ODBC and MSSQL. Ask on the DBIC list or on IRC, some guys use MSSQL successfully there. BR Alex > > -----Original Message----- > From: Alexander Hartmaier [mailto:alexander.hartma...@t-systems.at] > Sent: 02 July 2013 12:53 > To: catalyst@lists.scsys.co.uk > Subject: Re: [Catalyst] CSV / UTF-8 / Unicode > > On 2013-07-02 13:45, Craig Chant wrote: >> Hi Mike, >> >> The data is coming from MS SQL 2008 R2, I have checked the DB and the column >> is defined as data type nvarchar(max) , so is set to Unicode and when >> viewing data via SQL Studio Manager, the Unicode characters display fine. >> >> I am accessing the DB via DBI and have set on the connection string.. >> {'RaiseError' => 1, 'mysql_enable_utf8' => 1} > MSSQL != MySQL >> Though I'm not sure if the utf8 setting has any affect when using an MS ODBC >> driver as the flag implies it's for mysql ? > No it doesn't, read your DBD's docs. >> I tried to apply my $chars = decode('UTF-8', $xls); and I got an error >> saying you cannot decode a string with wide characters. >> >> I don't know what the string is, but when you try to decode you get a wide >> character error and when you try to encode you get a wide character error. >> >> If I don't do anything to it and just output it, no error, but the data is >> full of junk chars. >> >> Where do I go from here? > Ensure that the data coming out from your database is properly decoded, then > make sure it's properly encoded when outputting it. > Never mess with Perls' internal representation of the data by dis-/enabling > the utf-8 bit. >> Thanks, >> Craig. >> >> >> >> -----Original Message----- >> From: Mike Whitaker [mailto:m...@altrion.org] >> Sent: 02 July 2013 12:27 >> To: The elegant MVC web framework >> Subject: Re: [Catalyst] CSV / UTF-8 / Unicode >> >> >>> # encode UTF8 octet >>> use Encode qw(decode encode); >>> my $octets = encode('UTF-8', $xls); >> OUt of curiousity, where's the data in $xls come from, and what do you know >> about its encoding and bytes vs characters status? >> _______________________________________________ >> List: Catalyst@lists.scsys.co.uk >> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst >> Searchable archive: >> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ >> Dev site: http://dev.catalyst.perl.org/ This Email and any >> attachments contain confidential information and is intended solely for the >> individual to whom it is addressed. If this Email has been misdirected, >> please notify the author as soon as possible. If you are not the intended >> recipient you must not disclose, distribute, copy, print or rely on any of >> the information contained, and all copies must be deleted immediately. >> Whilst we take reasonable steps to try to identify any software viruses, any >> attachments to this e-mail may nevertheless contain viruses, which our >> anti-virus software has failed to identify. You should therefore carry out >> your own anti-virus checks before opening any documents. HomeLoan >> Partnership will not accept any liability for damage caused by computer >> viruses emanating from any attachment or other document supplied with this >> e-mail. HomeLoan Partnership reserves the right to monitor and archive all >> e-mail communications through its network. No representative or employee of >> HomeLoan Partnership has the authority to enter into any contract on behalf >> of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of >> H L Partnership Limited, registered in England and Wales with Registration >> Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L >> Partnership Limited is authorised and regulated by the Financial Conduct >> Authority. >> >> _______________________________________________ >> List: Catalyst@lists.scsys.co.uk >> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst >> Searchable archive: >> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ >> Dev site: http://dev.catalyst.perl.org/ > > > > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*" > *"*"*"* T-Systems Austria GesmbH Rennweg 97-99, 1030 Wien > Handelsgericht Wien, FN 79340b > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*" > *"*"*"* > Notice: This e-mail contains information that is confidential and may be > privileged. > If you are not the intended recipient, please notify the sender and then > delete this e-mail immediately. > *"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*"*" > *"*"*"* > > _______________________________________________ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ This Email and any attachments > contain confidential information and is intended solely for the individual to > whom it is addressed. If this Email has been misdirected, please notify the > author as soon as possible. If you are not the intended recipient you must > not disclose, distribute, copy, print or rely on any of the information > contained, and all copies must be deleted immediately. Whilst we take > reasonable steps to try to identify any software viruses, any attachments to > this e-mail may nevertheless contain viruses, which our anti-virus software > has failed to identify. You should therefore carry out your own anti-virus > checks before opening any documents. HomeLoan Partnership will not accept any > liability for damage caused by computer viruses emanating from any attachment > or other document supplied with this e-mail. HomeLoan Partnership reserves > the right to monitor and archive all e-mail communications through its > network. No representative or employee of HomeLoan Partnership has the > authority to enter into any contract on behalf of HomeLoan Partnership by > email. HomeLoan Partnership is a trading name of H L Partnership Limited, > registered in England and Wales with Registration Number 5011722. Registered > office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is > authorised and regulated by the Financial Conduct Authority. > > _______________________________________________ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ _______________________________________________ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ This Email and any attachments contain confidential information and is intended solely for the individual to whom it is addressed. If this Email has been misdirected, please notify the author as soon as possible. If you are not the intended recipient you must not disclose, distribute, copy, print or rely on any of the information contained, and all copies must be deleted immediately. Whilst we take reasonable steps to try to identify any software viruses, any attachments to this e-mail may nevertheless contain viruses, which our anti-virus software has failed to identify. You should therefore carry out your own anti-virus checks before opening any documents. HomeLoan Partnership will not accept any liability for damage caused by computer viruses emanating from any attachment or other document supplied with this e-mail. HomeLoan Partnership reserves the right to monitor and archive all e-mail communications through its network. No representative or employee of HomeLoan Partnership has the authority to enter into any contract on behalf of HomeLoan Partnership by email. HomeLoan Partnership is a trading name of H L Partnership Limited, registered in England and Wales with Registration Number 5011722. Registered office: 26-34 Old Street, London, EC1V 9QQ. H L Partnership Limited is authorised and regulated by the Financial Conduct Authority. _______________________________________________ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
