Re: [HACKERS] Making TEXT NUL-transparent

2011-12-23 Thread Florian Weimer
* Florian Pflug: On Nov24, 2011, at 10:54 , Florian Weimer wrote: Or is it not only about being able to *store* NULs in a text field? No, the entire core should be NUL-transparent. That's unlikely to happen. Yes, with the type input/output functions tied to NUL-terminated strings, that

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
Hello 2011/11/24 Florian Weimer fwei...@bfk.de: Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We take care of generating proper UTF-8, but this additional restriction sometimes slips by.  It would be really helpful if PostgreSQL could store such TEXT fields as-is (at

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Pavel Stehule: Hello 2011/11/24 Florian Weimer fwei...@bfk.de: Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We take care of generating proper UTF-8, but this additional restriction sometimes slips by.  It would be really helpful if PostgreSQL could store such TEXT

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Alexander Shulgin
Excerpts from Florian Weimer's message of Thu Nov 24 11:27:51 +0200 2011: and why you don't use bytea ? Text should be correct literal. It's actually UTF-8 text, and some PostgreSQL functions are only available for TEXT, but not BYTEA, e.g.: bfk_int= SELECT '\x006500'::bytea ~ 'A';

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
2011/11/24 Florian Weimer fwei...@bfk.de: * Pavel Stehule: Hello 2011/11/24 Florian Weimer fwei...@bfk.de: Occasionally, we get bitten by embedded NUL bytes in TEXT values.  We take care of generating proper UTF-8, but this additional restriction sometimes slips by.  It would be really

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Alexander Shulgin: It's actually UTF-8 text, and some PostgreSQL functions are only available for TEXT, but not BYTEA, e.g.: bfk_int= SELECT '\x006500'::bytea ~ 'A'; ERROR: operator does not exist: bytea ~ unknown And how will those TEXT functions behave on a value with an embedded

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
2011/11/24 Florian Weimer fwei...@bfk.de: * Alexander Shulgin: It's actually UTF-8 text, and some PostgreSQL functions are only available for TEXT, but not BYTEA, e.g.: bfk_int= SELECT '\x006500'::bytea ~ 'A'; ERROR:  operator does not exist: bytea ~ unknown And how will those TEXT

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Weimer
* Pavel Stehule: By the way, I refuse the notion that UTF-8 strings with embedded NULs are broken.  I can't recall any other system which enforces UTF-8 well-formedness, but does not permit embedded NULs. I have a different question. What is reason for embedded NULs inside strings? The

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Alexander Shulgin
Excerpts from Florian Weimer's message of Thu Nov 24 12:59:09 +0200 2011: I have a different question. What is reason for embedded NULs inside strings? The source system does not enforce that constraint, so from time to time, such data slips through. I don't know why it's there in the

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Pavel Stehule
2011/11/24 Alexander Shulgin a...@commandprompt.com: Excerpts from Florian Weimer's message of Thu Nov 24 12:59:09 +0200 2011: I have a different question. What is reason for embedded NULs inside strings? The source system does not enforce that constraint, so from time to time, such data

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Robert Haas
On Thu, Nov 24, 2011 at 4:54 AM, Florian Weimer fwei...@bfk.de wrote: By the way, I refuse the notion that UTF-8 strings with embedded NULs are broken.  I can't recall any other system which enforces UTF-8 well-formedness, but does not permit embedded NULs. This seems like a key point. If

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Florian Pflug
On Nov24, 2011, at 10:54 , Florian Weimer wrote: Or is it not only about being able to *store* NULs in a text field? No, the entire core should be NUL-transparent. That's unlikely to happen. A more realistic approach would be to solve this only for UTF-8 encoded strings by encoding the NUL

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Andrew Dunstan
On Thu, Nov 24, 2011 at 4:54 AM, Florian Weimer fwei...@bfk.de wrote: By the way, I refuse the notion that UTF-8 strings with embedded NULs are broken. I can't recall any other system which enforces UTF-8 well-formedness, but does not permit embedded NULs. Refuse away, but I don't think

Re: [HACKERS] Making TEXT NUL-transparent

2011-11-24 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On Thu, Nov 24, 2011 at 4:54 AM, Florian Weimer fwei...@bfk.de wrote: By the way, I refuse the notion that UTF-8 strings with embedded NULs are broken. I can't recall any other system which enforces UTF-8 well-formedness, but does not permit embedded