Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Chris Withers
On 21/09/2010 23:59, Marius Gedminas wrote:
 On Tue, Sep 21, 2010 at 07:39:46PM +0100, Chris Withers wrote:
 So, would anyone object if I change the default, on both trunk and the
 2.12 branch to some variant of:

 class BasicEncodingConflictResolver(object):
implements(IUnicodeEncodingConflictResolver)

def resolve(self, context, text, expression):
logging.warn('You should register an '
 'IUnicodeEncodingConflictResolver that matches '
 'your content')
return text.decode('ascii','replace')

 ...and delete that devil spawn turd that is PreferredCharsetResolver?

 If this means Zope2-the-application becomes unusable out of the box if
 you've non-ASCII data in your database, then I'm -1.

I'd call having the odd '?' shown in pages and a warning in your logs 
telling you about the problem in clear terms versus sporadic total 
failures by way of unicode exceptions a lot more usable, rather than the 
current situation, which is conceptually insane *and* unusable ;-)

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Chris Withers
On 22/09/2010 03:50, Andreas Jung wrote:
 I also strongly object such a change in the default behavior.

I'm sorry, but WHAT?

 I did a
 lot of testing of the current implementation and configuration with
 various browsers and Zope applications and add-ons in order to make the
 Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.

Er, you can't be serious...

The idea of asking the user agent what encoding is used to encode data 
in the database is like telling an obese person they can have their foot 
amputated in order to lose weight.

A less insane version of that default would be:

class BasicEncodingConflictResolver(object):
   implements(IUnicodeEncodingConflictResolver)

   def resolve(self, context, text, expression):
   logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
   encodings = ['utf-8','latin-1']
   mpc =  getattr(context,'management_page_charset',None)
   if mpc:
   encodings.insert(0,mpc)
   for enc in encodings:
   try:
   return unicode(text,enc)
   except UnicodeDecorError:
   pass
   return unicode(text,sys.getdefaultencoding(),'replace')


 However with the given constraints there will be always applications or
 bad written code

PreferredCharsetResolver *is* bad code.

I'll say it again, 'cos the message doesn't seem to be getting through:

What on earth do the character sets accepted by the browser have to do 
with the python string encoding used in the data on the server, which up 
until that request, most likely has never been anywhere near that 
browser before?!

 resolver. That's why it is pluggable.

Yes, and well documented and advertised too... not!

 existing resolver smarter and more efficient: +1 for tuning the
 implementation but the proposed solution above does not solve anything
 and will break a bunch of apps.

bull crap.

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Charlie Clark
Am 22.09.2010, 10:24 Uhr, schrieb Chris Withers ch...@simplistix.co.uk:

 I'd call having the odd '?' shown in pages and a warning in your logs
 telling you about the problem in clear terms versus sporadic total
 failures by way of unicode exceptions a lot more usable, rather than the
 current situation, which is conceptually insane *and* unusable

Next Wednesday is bug day - a good chance for a concerted operation on  
this. Chris, your description that the current implementation contravenes  
interface specification means that it is a bug. But no change should  
simply break existing applications. Can we come up with some tests (ZODB  
data and presumably client encodings) and a workaround to enable the  
fudge? I don't know how much non-ASCII you have but the odd ? might turn  
out to be quite a lot for some people.

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting  Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Chris Withers
On 21/09/2010 21:18, Andreas Jung wrote:
 I can not recall all details but at the time of the integration of the
 Zope 3 ZPT engine the unicode resolver approach was the best thinkable
 solution for dealing all possible edgecase where encoding issues may pop
 *for the sake of backwards compatibility*

...where it miserably failed. The projects I've hit this on have all 
been running flawlessly on Zope 2.1(ish) through to Zope 2.9.
Zope 2.11 onwards, in my experience, phail, spectacularly...

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Andreas Jung
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chris Withers wrote:
 On 22/09/2010 03:50, Andreas Jung wrote:
 I also strongly object such a change in the default behavior.
 
 I'm sorry, but WHAT?
 
 I did a
 lot of testing of the current implementation and configuration with
 various browsers and Zope applications and add-ons in order to make the
 Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.
 
 Er, you can't be serious...
 
 The idea of asking the user agent what encoding is used to encode data
 in the database is like telling an obese person they can have their foot
 amputated in order to lose weight.
 
 A less insane version of that default would be:
 
 class BasicEncodingConflictResolver(object):
   implements(IUnicodeEncodingConflictResolver)
 
   def resolve(self, context, text, expression):
   logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
   encodings = ['utf-8','latin-1']
   mpc =  getattr(context,'management_page_charset',None)
   if mpc:
   encodings.insert(0,mpc)
   for enc in encodings:
   try:
   return unicode(text,enc)
   except UnicodeDecorError:
   pass
   return unicode(text,sys.getdefaultencoding(),'replace')
 
 
 However with the given constraints there will be always applications or
 bad written code
 
 PreferredCharsetResolver *is* bad code.
 
 I'll say it again, 'cos the message doesn't seem to be getting through:
 
 What on earth do the character sets accepted by the browser have to do
 with the python string encoding used in the data on the server, which up
 until that request, most likely has never been anywhere near that
 browser before?!
 
 resolver. That's why it is pluggable.
 
 Yes, and well documented and advertised too... not!
 
 existing resolver smarter and more efficient: +1 for tuning the
 implementation but the proposed solution above does not solve anything
 and will break a bunch of apps.
 
 bull crap.
 
 Chris

Please calm down - I am not up to reply to a posting written in a tone
that is even far beyond my personal style when ranting on other peoples
crap.

As said: the discussion is documented in some Launchpad ticket.


If you think it is crap then I cam happy to accept that (and I don't
mind much because Zope 2 is not my radar right now). You are committer
and you can fix my crap. Blame me for having written this crap but you
could also blame me for having on this whole Zope 3 ZPT engine
integration :-)

Cheers,
Andreas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJMmcU5AAoJEADcfz7u4AZj1wMLvRQ9riRMebLtqZe4VY9U0SCg
v8S8uS/gZXvBDlDodoNoPurumWGlabCXORD5GRgVZhGiSub1GmTXqslHD5Hro1xf
FeGr3skMlTlAis9xjH3qUuBUbivk5+ebk75ii/jPKJmxRg87a3Gwhi1eFnnCZBTg
AGWzYu5QFmKI/iHgWpYjdQiJ1fS2MtRrisF6JCVuduBJs3kenHAV8huw8/7azv6l
uI/tUQjg2N49s14NWZbIObe/cDY6SkSoiu4ifINTJX68aJxgywVqzGu1aiqR33t3
/j2v347/4TCwdfJGFBopvHLBdoI0WIHA4DB9C2VMuAYO/eOQSrH84gGLqbWeRBku
tfh7KqLa5mulyufXVc3YtgF14rk2kJ5TvRohM4YuR1RFB/0WGU6p0FzD8KZ23pB+
CXk9puJi/FTsIO+eSgDcuPZhoc6ZJoPgA2s03TD+8ghspnWSXU4hbZ2DncYzlDOu
Ss+ri+thcf5APSrvfCb4B7xLq/tW2do=
=yS8z
-END PGP SIGNATURE-
attachment: lists.vcf___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Chris Withers
On 22/09/2010 09:58, Andreas Jung wrote:
 If you think it is crap then I cam happy to accept that (and I don't
 mind much because Zope 2 is not my radar right now). You are committer
 and you can fix my crap. Blame me for having written this crap but you
 could also blame me for having on this whole Zope 3 ZPT engine
 integration :-)

Can you please answer me one thing though: in what way could it ever be 
a good idea to consult the browser's list of acceptable character sets 
to decide how to decode python strings into python unicode objects in a 
database that the browser has never had anything to do with?

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Andreas Jung
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chris Withers wrote:
 On 21/09/2010 21:18, Andreas Jung wrote:
 I can not recall all details but at the time of the integration of the
 Zope 3 ZPT engine the unicode resolver approach was the best thinkable
 solution for dealing all possible edgecase where encoding issues may pop
 *for the sake of backwards compatibility*
 
 ...where it miserably failed. The projects I've hit this on have all 
 been running flawlessly on Zope 2.1(ish) through to Zope 2.9.
 Zope 2.11 onwards, in my experience, phail, spectacularly...

Sorry, not my problem!

The Zope 3 ZPT engine together with the resolver is now available for at
least two or three yearswe had a *very long* beta phase for the ZPT
integration..have you tested? Obviously no...blaming me for doing a lot
of testing while having seen zero contributions in terms of code and
testing in this particular case is laughable.  Such allegations were one
of the reasons for quitting as Zope 2 release manager. If the release
manager is the idiot doing the dirty work and taking the beating from
people not contributing to the project that I was happy being the Zope 2
idiot for years but I am no longer interested dealing with a minority of
people knowing better and constantly playing the niggler.

Andreas

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJMmcdSAAoJEADcfz7u4AZjn+4LvA2JtT++GJD+taeZZ+B95UGt
fB7fINT1nUyXNWxDV6OyoZd2zgJoCX2Yg3V3FibVaz5uiQjGNkZnPvLooLP+9ESn
lSyc6D7tYgHzPhNU/OitwUp8pjZeOWhLXShR0PH1TVtv9I8PAx/0NRDNZDLpE9Cx
v1TbTlRztY+crRDkGK4xf9runTNZwVw5yyBk6NJXnKr1yMIOhZ72Fz0LAReiIHq0
vkgSakS7vhYde2acsPi4lXC4lS6s7T67/GjbcPWQAtUw4f4Xc4linYERvFCHUYOT
w4z2CDahhyu7ZTfzguGCuUEQZAffi9OlY5LbFlXMSF9crFjrCrqxvPhGmAQXN5O+
z6tYlAXGCZpuV3gMNIPjJBej6UujYP/mnjT8SC5BYkGlYrriNfoBttVynK5TJpMk
z5s90fDZa4hETaIXAxTkpCiBfX0IFUp7aqb+lU7HsSs4rHj331cknbemX/iq+S7O
ohb5SMgDqJg20UTxxNFH/2ZLSdWsOm8=
=Wyxr
-END PGP SIGNATURE-
attachment: lists.vcf___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Andreas Jung
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chris Withers wrote:
 On 22/09/2010 09:58, Andreas Jung wrote:
 If you think it is crap then I cam happy to accept that (and I don't
 mind much because Zope 2 is not my radar right now). You are committer
 and you can fix my crap. Blame me for having written this crap but you
 could also blame me for having on this whole Zope 3 ZPT engine
 integration :-)
 
 Can you please answer me one thing though: in what way could it ever be
 a good idea to consult the browser's list of acceptable character sets
 to decide how to decode python strings into python unicode objects in a
 database that the browser has never had anything to do with?

Check with Launchpad - neither I can remember the details nor am I
interested right now thinking about. There were reasons *and* real-world
tests with browsers obviously required that. As said:
the encoding handling is a game you can only lose inside Zope 2.

- -aj
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJMmcfoAAoJEADcfz7u4AZjQ90LwLQoiWZzKwnLhYJBaO/j4Hs5
RXxURlWAPX2VB1iNbzmzZgadffYdF4Tigmh/kuZFZwZLizJw+SnYwPwJJgAHM/dn
aq2Q9JV3atjlu0p/OGe/54BY5ZfMWSE/AjS2gKSJExNdp9rKZbvOdKWg/J+utzq3
M/U7E2AXvOtG3YWZYggQF1QZlY3PZu08XlXHUCNdiK3fAu9dtJQstRGGdggdxhuQ
QiheKAs2Tu/hKWsVPURlxes9zGtylpDAQxt5ycRxTxYToJyCQtFdSzBsbLC1rkGG
fMDOOL/fPR3UlTzyhvB87ugaiBH7owng+n8tia3Ro06LzGp0oy5PdytrhfQ2BGn2
RzPre7tGavUFn6sxYSEZ6akIuHwk1KQgvMFN2ngsa/F1N1xqZpQUXwF1ALrEFzWc
qg4CeO4UDdkh1V8ZlawbKgbih/0eaNYKreCg0vCb2xNqXvsl5OL0QERj0v4u/gWp
pJ1zGB+BVHbtZifhHSHkKYmx9Dgpl3I=
=EwM7
-END PGP SIGNATURE-
attachment: lists.vcf___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Marius Gedminas
On Wed, Sep 22, 2010 at 09:34:59AM +0100, Chris Withers wrote:
 On 22/09/2010 03:50, Andreas Jung wrote:
  I also strongly object such a change in the default behavior.
 
 I'm sorry, but WHAT?
 
  I did a
  lot of testing of the current implementation and configuration with
  various browsers and Zope applications and add-ons in order to make the
  Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.

Did various browsers include Safari and MSIE, I wonder?  Because those
are the ones causing me pain and UnicodeDecodeErrors.

 A less insane version of that default would be:
 
 class BasicEncodingConflictResolver(object):
implements(IUnicodeEncodingConflictResolver)
 
def resolve(self, context, text, expression):
logging.warn('You should register an '
 'IUnicodeEncodingConflictResolver that matches '
 'your content')
encodings = ['utf-8','latin-1']
mpc =  getattr(context,'management_page_charset',None)
if mpc:
encodings.insert(0,mpc)
for enc in encodings:
try:
return unicode(text,enc)
except UnicodeDecorError:
pass
return unicode(text,sys.getdefaultencoding(),'replace')

I would prefer this to current implementation.

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3/BlueBream consulting and development


signature.asc
Description: Digital signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-22 Thread Marius Gedminas
On Wed, Sep 22, 2010 at 09:24:11AM +0100, Chris Withers wrote:
 On 21/09/2010 23:59, Marius Gedminas wrote:
  If this means Zope2-the-application becomes unusable out of the box if
  you've non-ASCII data in your database, then I'm -1.
 
 I'd call having the odd '?' shown in pages and a warning in your logs 
 telling you about the problem in clear terms versus sporadic total 
 failures by way of unicode exceptions a lot more usable, rather than the 
 current situation, which is conceptually insane *and* unusable ;-)

There are people who install Zope 2 and then build their websites
through the web exclusively.  Asking them to figure out how to write and
register a utility when they upgrade their 2.12 Zope site to 2.13 is a
bit too much, in my opinion.

Why replace a bad situation with a different bad situation?  How would
you like a piece of software that r?pl?c?d ?ll th? v?w?ls w?th q??st??n
m?rks ??t ?f th? b?x unless you did a highly technical voodoo dance
under the hood?

I would be happy with the other version you pasted in this thread, the
one that tries context.management_page_charset, UTF-8, and Latin-1.
Let's do that instead.

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3/BlueBream consulting and development


signature.asc
Description: Digital signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Chris Withers
Hi All,

It's been a a while since I had a good rant on a Zope list, but this 
really takes the biscuit.

Andreas, what on *earth* were you thinking with PreferredCharsetResolver?!

I like the idea of IUnicodeEncodingConflictResolver, but 
PreferredCharsetResolver is in the realms of totally batshit crazy.

Why? well, because what *on earth* have the character sets accepted by 
the user agent got to do with how strings are encoded in *my database*?!

Nevermind the logic holes of the following:

- reverting to management_page_charset *only* if there's no REQUEST 
(there's always a REQUEST, this is Zope 2, the world collapses if there 
isn't!)

- returning the original text if the bizarre dance fails, violating the 
contract of IUnicodeEncodingConflictResolver to *return unicode*

How about the *insane performance overhead* of doing two utility lookups 
*and* iterating over a load of encodings trying to find one that works 
for *every single string substituition* done by a ZPT?!

Why has no-one noticed this? Well, my guess is because when browsers are 
explicitly supplying headers, as quite a few do (IE and Safari excepted, 
who are legitimately going we can handle anything you can throw at 
us), this moronic piece of code happily loops through them all, and no 
doubt gets a hit on either utf-8 or latin-1, or, if you're really lucky, 
ascii.

To boot, when things go wrong, nobody suspects this miserable little 
turd because it's hides itself nicely by just returning the original 
text, leaving the bemused reader to wonder why some UA's fail and some 
succeed, pointing the finger in totally the wrong direction (hence 
Charlie's hacked up getPreferredCharsets and poor Vlad's desperate 
attempts).

Thankfully, I guess I can register an override that doesn't bark at the 
moon and froth at the mouth... Here, have an invoice for the hours of my 
life you've needlessly wasted...

Farq,

Chris ;-)

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
 - http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Charlie Clark
Am 21.09.2010, 20:17 Uhr, schrieb Chris Withers ch...@simplistix.co.uk:

 To boot, when things go wrong, nobody suspects this miserable little
 turd because it's hides itself nicely by just returning the original
 text, leaving the bemused reader to wonder why some UA's fail and some
 succeed, pointing the finger in totally the wrong direction (hence
 Charlie's hacked up getPreferredCharsets and poor Vlad's desperate
 attempts).

Chris,

rant aside it sounds like you've identified a bug. Can you submit a bug so  
we can hopefully fix it in 2.13?

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting  Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Hanno Schlichting
On Tue, Sep 21, 2010 at 8:17 PM, Chris Withers ch...@simplistix.co.uk wrote:
 I like the idea of IUnicodeEncodingConflictResolver, ...

[...]

 Why has no-one noticed this?

Because there's not all that many people left using plain Zope 2 and
upgrading to new versions.

The rest is using Plone, which has its own version of this:

def _unicode_replace(structure):
if isinstance(structure, str):
text = structure.decode('utf-8', 'replace')
else:
text = unicode(structure)
return text

class UTF8EncodingConflictResolver(object):
implements(IUnicodeEncodingConflictResolver)

def resolve(self, context, text, expression):
return _unicode_replace(text)

We can do this, as we force manage_page_charset and
default-zpublisher-encoding to be utf-8 and only support utf-8 and
Unicode in the database.

Zope 2 itself tries to support arbitrary encodings, at which point it
can only guess badly or fail.

Hanno
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Chris Withers
On 21/09/2010 19:32, Hanno Schlichting wrote:
 Because there's not all that many people left using plain Zope 2 and
 upgrading to new versions.

Well, or they assume they must be doing something wrong because of this 
default.

 The rest is using Plone, which has its own version of this:

 def _unicode_replace(structure):
  if isinstance(structure, str):
  text = structure.decode('utf-8', 'replace')
  else:
  text = unicode(structure)
  return text

 class UTF8EncodingConflictResolver(object):
  implements(IUnicodeEncodingConflictResolver)

  def resolve(self, context, text, expression):
  return _unicode_replace(text)

Right, although the 'replace' is a lame cop out too ;-)

 We can do this, as we force manage_page_charset and
 default-zpublisher-encoding to be utf-8 and only support utf-8 and
 Unicode in the database.

Fundamentally, the developer must know what encoding is being used in 
their data. So, they should be able to register an appropriate decoder 
BUT...

 Zope 2 itself tries to support arbitrary encodings, at which point it
 can only guess badly or fail.

...having this default, which makes ABSOLUTELY NO SENSE in this 
universe, or likely any other, is fucking crazy.

So, would anyone object if I change the default, on both trunk and the 
2.12 branch to some variant of:

class BasicEncodingConflictResolver(object):
  implements(IUnicodeEncodingConflictResolver)

  def resolve(self, context, text, expression):
  logging.warn('You should register an '
   'IUnicodeEncodingConflictResolver that matches '
   'your content')
  return text.decode('ascii','replace')

...and delete that devil spawn turd that is PreferredCharsetResolver?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing  Python Consulting
- http://www.simplistix.co.uk
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Charlie Clark
Am 21.09.2010, 20:39 Uhr, schrieb Chris Withers ch...@simplistix.co.uk:

 So, would anyone object if I change the default, on both trunk and the
 2.12 branch to some variant of:
 class BasicEncodingConflictResolver(object):
   implements(IUnicodeEncodingConflictResolver)
  def resolve(self, context, text, expression):
   logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
   return text.decode('ascii','replace')

This sounds like a very good idea. Might benefit from some brief  
documentation, to facilitate testing.

Charlie
-- 
Charlie Clark
Managing Director
Clark Consulting  Research
German Office
Helmholtzstr. 20
Düsseldorf
D- 40215
Tel: +49-211-600-3657
Mobile: +49-178-782-6226
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Andreas Jung
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

First, cool down for a moment.

Second, I discussed all those issues with my special friend in some
Launchpad ticket years ago (+ a huge amount of private discussions).
I can not recall all details but at the time of the integration of the
Zope 3 ZPT engine the unicode resolver approach was the best thinkable
solution for dealing all possible edgecase where encoding issues may pop
*for the sake of backwards compatibility* and with all possible
combinations where encodings have to be considered (including stuff like
the management_page_charset property). The implementation is
now working for years - so it can't be that bad. If you have a better
implementation you should be able to override the utility or to fix
the code directly.

I won't comment after years on the details below (check Launchpad for
the related ticket).

Andreas



Chris Withers wrote:
 Hi All,
 
 It's been a a while since I had a good rant on a Zope list, but this
 really takes the biscuit.
 
 Andreas, what on *earth* were you thinking with PreferredCharsetResolver?!
 
 I like the idea of IUnicodeEncodingConflictResolver, but
 PreferredCharsetResolver is in the realms of totally batshit crazy.
 
 Why? well, because what *on earth* have the character sets accepted by
 the user agent got to do with how strings are encoded in *my database*?!
 
 Nevermind the logic holes of the following:
 
 - reverting to management_page_charset *only* if there's no REQUEST
 (there's always a REQUEST, this is Zope 2, the world collapses if there
 isn't!)
 
 - returning the original text if the bizarre dance fails, violating the
 contract of IUnicodeEncodingConflictResolver to *return unicode*
 
 How about the *insane performance overhead* of doing two utility lookups
 *and* iterating over a load of encodings trying to find one that works
 for *every single string substituition* done by a ZPT?!
 
 Why has no-one noticed this? Well, my guess is because when browsers are
 explicitly supplying headers, as quite a few do (IE and Safari excepted,
 who are legitimately going we can handle anything you can throw at
 us), this moronic piece of code happily loops through them all, and no
 doubt gets a hit on either utf-8 or latin-1, or, if you're really lucky,
 ascii.
 
 To boot, when things go wrong, nobody suspects this miserable little
 turd because it's hides itself nicely by just returning the original
 text, leaving the bemused reader to wonder why some UA's fail and some
 succeed, pointing the finger in totally the wrong direction (hence
 Charlie's hacked up getPreferredCharsets and poor Vlad's desperate
 attempts).
 
 Thankfully, I guess I can register an override that doesn't bark at the
 moon and froth at the mouth... Here, have an invoice for the hours of my
 life you've needlessly wasted...
 
 Farq,
 
 Chris ;-)
 


- -- 
ZOPYX Limited   | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope  Plone
D-72070 Tübingen| Produce  Publish
www.zopyx.com   | www.produce-and-publish.com
- 
E-Publishing, Python, Zope  Plone development, Consulting


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJMmRMqAAoJEADcfz7u4AZjThgLvjIWRhiVWoV/0IG7qadEliG0
Dk1+v+hu3ocu0DURn9XK5nGrHkkR+cFIjYdyjArDfXTqoNXWPii90QbtypK7Lce8
PD2rPp+fnwyMSne9iz9s/LtYYrDO/gbxWx9robjyjGmvICthKZD/qZk3cM47SSzq
rhZLRV+r6GwVemn8sQEQ/y3gXKMgP7txz/Co1reWdWgrRH2HX4qpzvKHVLykXIrT
lx4D9X0UfxGO+64p7MCEOgg41W3EzNM3qIEGp4PUTTwjcp/Use+AP5kS+Q6vQEWV
n4c3RdaIX35EMNjVaunh0SMj+75rizMTarkGJ9e5lxekQLxqrveTiKqgGWtManD4
fjCSXpaobvy1Ym02CSLSLmSA4Y8ETTvdM87covKBvXHfaDyn5zxZaCmJ66L7Y+1i
OwnbtFIke4tLFGhsEQDcbSBQDMD6TNuA1h/SQJ5AXAT3kSBPPvhWS/C8tWvoEdft
dhzLEvXmpqMyA/4EgrMh6zP5fuGGkuM=
=WUsF
-END PGP SIGNATURE-
attachment: lists.vcf___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Marius Gedminas
On Tue, Sep 21, 2010 at 07:39:46PM +0100, Chris Withers wrote:
 So, would anyone object if I change the default, on both trunk and the 
 2.12 branch to some variant of:
 
 class BasicEncodingConflictResolver(object):
   implements(IUnicodeEncodingConflictResolver)
 
   def resolve(self, context, text, expression):
   logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
   return text.decode('ascii','replace')
 
 ...and delete that devil spawn turd that is PreferredCharsetResolver?

If this means Zope2-the-application becomes unusable out of the box if
you've non-ASCII data in your database, then I'm -1.

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3/BlueBream consulting and development


signature.asc
Description: Digital signature
___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] IUnicodeEncodingConflictResolver moronosity

2010-09-21 Thread Andreas Jung
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Marius Gedminas wrote:
 On Tue, Sep 21, 2010 at 07:39:46PM +0100, Chris Withers wrote:
 So, would anyone object if I change the default, on both trunk and the 
 2.12 branch to some variant of:

 class BasicEncodingConflictResolver(object):
   implements(IUnicodeEncodingConflictResolver)

   def resolve(self, context, text, expression):
   logging.warn('You should register an '
'IUnicodeEncodingConflictResolver that matches '
'your content')
   return text.decode('ascii','replace')

 ...and delete that devil spawn turd that is PreferredCharsetResolver?
 
 If this means Zope2-the-application becomes unusable out of the box if
 you've non-ASCII data in your database, then I'm -1.
 


I also strongly object such a change in the default behavior. I did a
lot of testing of the current implementation and configuration with
various browsers and Zope applications and add-ons in order to make the
Zope 3 ZPT engine in Zope 2 work out-of-the-box for most applications.
However with the given constraints there will be always applications or
bad written code or specific requirements were you need to change the
resolver. That's why it is pluggable. There is is room for making the
existing resolver smarter and more efficient: +1 for tuning the
implementation but the proposed solution above does not solve anything
and will break a bunch of apps.

Andreas
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJMmW78AAoJEADcfz7u4AZjs5MLv1TK/Y4gzX3SgnE0//ZPgx2g
FSMcLkli2AT2//83U6w9NTDxre4OEkpfCZVaQbMGAdoooCPBcgLCV1Mvd44aVI7Q
f6ff+1B22d+BxDEIJZOU05hFjJgxh4EQE7i8/wTVAoz+vq/HVbppXV5btu6Cyy9+
RsjdIUaqqIJ1FPDPAIG7BF+c2h+amcPZjoErN5/vxc4HSkNU4vOUG67bCh4Ea8eL
6RCiBT1Hr3zkNYGsdwc7m9fsTjz3jAi2o20qWCk0xeYaz0SQxvAkSPMrvbz3fL+P
GMAaP5tU/ZBbUgIGD6OfUlrx1uaTeoNKRP1tA1FavQPXYaIuPUVoTXhV9c/5gp9h
zZQbFQofSv8hX7PfDp3kGCiA11lBk6XAAiB7nO6PqdC/uDD1pqMuIa6h2Gzo71B6
kTTrKs2M9m7IKTwuq58SkGmJu7zUchnn7fcgXUjxv/T7z52tyN/rMemp5YaxG7O3
644kz3G4zelQpLeH5TOZwmaBcSWJ10g=
=ZPTw
-END PGP SIGNATURE-
attachment: lists.vcf___
Zope-Dev maillist  -  Zope-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope )