Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-08 Thread Oliver Jowett
Madison Kelly wrote: Is there a way to store the name in raw binary? Yes: bytea. -O ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-08 Thread Karsten Hilbert
> There is a table t1(member_id integer primary key, member_name text, > address text, phone text, email text). I have to reach each member by > either adress, phone or email. Unfortunately some of address field > have wrong encoded data. In this case I will use phone or email to > reach them. >

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-08 Thread Karsten Hilbert
> Is there a way to store the name in raw binary? If so, would this not > be safe because to postgresql it should no longer matter what data is or > represents, right? Maybe there is a third option I am not yet concidering? In the backup rename the file and add another file .README which expla

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-08 Thread John Hansen
Tatsuo Ishii wrote: > Sent: Sunday, May 08, 2005 3:31 PM > To: John Hansen > Cc: [EMAIL PROTECTED]; pgsql-general@postgresql.org; > pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem > > > Tatsuo Ishii wrote: > > > Se

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-08 Thread Tino Wildenhain
Am Sonntag, den 08.05.2005, 14:30 +0900 schrieb Tatsuo Ishii: ... > Actually I myself thought as you are before. Later I found that it was > not so good idea. People already have invalid encoded data in their > precious database and have very hard time to migrate to newer version > of PostgreSQL be

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread Tatsuo Ishii
> Tatsuo Ishii wrote: > > Sent: Sunday, May 08, 2005 12:01 PM > > To: [EMAIL PROTECTED] > > Cc: pgsql-general@postgresql.org; pgsql-hackers@postgresql.org > > Subject: Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem > > > > We have develo

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread Tom Lane
"John Hansen" <[EMAIL PROTECTED]> writes: > Tatsuo Ishii wrote: >> We have developed patches which relaxes the character >> validation so that PostgreSQL accepts invalid characters. > That is just plain 100% wrong!! That was my first reaction too. Why would this be a good idea? If someone does

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread John Hansen
Madison Kelly wrote: >Under most circumstances I would agree with you completely. In my > case though I have to decide between risking a loss of a > user's data or > attempt to store the file name in some manner that would > return the same > name used by the file system. > >The user

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread Madison Kelly
John Hansen wrote: Tatsuo Ishii wrote: We have developed patches which relaxes the character validation so that PostgreSQL accepts invalid characters. It works like this: That is just plain 100% wrong!! Under no circumstances should there be invalid data in a database. And if you're trying to ma

Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread John Hansen
Tatsuo Ishii wrote: > Sent: Sunday, May 08, 2005 12:01 PM > To: [EMAIL PROTECTED] > Cc: pgsql-general@postgresql.org; pgsql-hackers@postgresql.org > Subject: Re: [HACKERS] [GENERAL] Invalid unicode in COPY problem > > We have developed patches which relaxes the character &g

Re: [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread Madison Kelly
Thank you, I would! What versions have you tested the patch against? I am sorry but I am not too familiar with applying patches against the main program, is there documentation on how to apply the patch? Is there a way to roll the patch back/remove it? Would I be able to script th

Re: [GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread Tatsuo Ishii
We have developed patches which relaxes the character validation so that PostgreSQL accepts invalid characters. It works like this: 1) new postgresql.conf item "mbstr_check" added. 2) if mbstr_check = 0 then invalid characters are not accepted (same as current PostgreSQL behavior). This is the

[GENERAL] Invalid unicode in COPY problem

2005-05-07 Thread Madison Kelly
Hi all, I've been chasing down a bug and from what I have learned it may be because of how postgreSQL (8.0.2 on Fedora Core 4 test 2) handles invalid unicode. I've been given some ideas on how to try to catch invalid unicode but it seems expensive so I am hoping there is a postgresql way to d