Bruce Momjian wrote: > Tom Lane wrote: > > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > > I have added your suggestions to the 8.1.X release notes. > > > > Did you read the followup discussion? Recommending -c without a large > > warning seems a very bad idea. > > Well, I said it would remove invalid sequences. What else should we > say? > > This will remove invalid character sequences. > > I saw no clear solution that allowed sequences to be corrected.
The release note text is: Some users are having problems loading <literal>UTF8</> data into 8.1.X. This is because previous versions allowed invalid <literal>UTF8</> sequences to be entered into the database, and this release properly accepts only valid <literal>UTF8</> sequences. One way to correct a dumpfile is to use <command>iconv -c -f UTF-8 -t UTF-8</>. This will remove invalid character sequences. <command>iconv</> reads the entire input file into memory so it might be necessary to <command>split</> the dump into multiple smaller files for processing. One nice solution would be if iconv would report the lines with errors and you could correct them, but I see no way to do that. The only thing you could do is to diff the old and new files to see the problems. Is that helpful? Here is new text I have used: Some users are having problems loading <literal>UTF8</> data into 8.1.X. This is because previous versions allowed invalid <literal>UTF8</> sequences to be entered into the database, and this release properly accepts only valid <literal>UTF8</> sequences. One way to correct a dumpfile is to use <command>iconv -c -f UTF-8 -t UTF-8 -o cleanfile.sql dumpfile.sql</>. The <literal>-c</> option removes invalid character sequences. A diff of the two files will show the sequences that are invalid. <command>iconv</> reads the entire input file into memory so it might be necessary to <command>split</> the dump into multiple smaller files for processing. It highlights the 'diff' idea. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania 19073 ---------------------------(end of broadcast)--------------------------- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match