Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Sunday 26 January 2003 7:25 pm, Kazuya FUKAMACHI wrote: Have you decided which version the patch should be included? I combined 4 of the patches and Yousei Tahara's patch into single patch and published for japanese users on Jan 15. http://www.atransia.co.jp/home/ZenKai/Members/kafka/patch/Zope261b.patch Thanks to everyone who has helped on this. This patch is in cvs ready for 2.6.1 beta 2. Please give the beta a good work out. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Sunday 26 January 2003 7:25 pm, Kazuya FUKAMACHI wrote: On Tue, 14 Jan 2003 10:33:40 + Toby Dickenson [EMAIL PROTECTED] wrote: I am currently looking at getting this into 2.6.1 or 2.6.2. I would appreciate confirmation that you are all happy with this combination of patches. Have you decided which version the patch should be included? I am trying to get confirmation of the release schedule for 2.6.1 beta 2. If only I had known it would be delayed this long back in December... I've announced the patch by Japanese Zope Users Group Mailing list, which has more than 1000 subscribers. More than 100 people downloaded it, and any no complaints heard. Many said they were happy with the patch. Thats good news. Thankyou. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tue, 14 Jan 2003 10:33:40 + Toby Dickenson [EMAIL PROTECTED] wrote: I am currently looking at getting this into 2.6.1 or 2.6.2. I would appreciate confirmation that you are all happy with this combination of patches. Have you decided which version the patch should be included? I've announced the patch by Japanese Zope Users Group Mailing list, which has more than 1000 subscribers. More than 100 people downloaded it, and any no complaints heard. Many said they were happy with the patch. I combined 4 of the patches and Yousei Tahara's patch into single patch and published for japanese users on Jan 15. http://www.atransia.co.jp/home/ZenKai/Members/kafka/patch/Zope261b.patch So, I could count the number of downloads ;) Thanks, Kazuya Fukamachi ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Saturday 21 December 2002 10:54 pm, Kazuya FUKAMACHI wrote: I applied Toby's patch + Yousei's patch, and it works. Thanks to your neat piece of work, I can start to work on Zope 2.6.0 now, and go farther. I am currently looking at getting this into 2.6.1 or 2.6.2. I would appreciate confirmation that you are all happy with this combination of patches. thanks, -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi. I am currently looking at getting this into 2.6.1 or 2.6.2. I would appreciate confirmation that you are all happy with this combination of patches. Thank you:-) BTW, I was asked for upload my patch on collector. Could you put it in a issue 737 page? I don't permit to upload it. Regards. -- Yusei ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Friday 20 December 2002 6:26 pm, Yusei TAHARA wrote: 1. browsers autodetection make a mistake. Not a browser mistake - this is *my* mistake. thanks. 2. management_page_charset_tag really needs? Yes, I agree with your analysis. I made a patch for fix those problems. It looks good work on my zope. Good work. This definitely looks viable for 2.6.2 -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi, On Sat, 21 Dec 2002 03:26:49 +0900 Yusei TAHARA [EMAIL PROTECTED] wrote: I tried to your patch 03-properties.diff, then I found two problems. 1. browsers autodetection make a mistake. 2. management_page_charset_tag really needs? I applied Toby's patch + Yousei's patch, and it works. Thanks to your neat piece of work, I can start to work on Zope 2.6.0 now, and go farther. Regards Kazuya Fukamachi ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Wednesday 11 December 2002 2:15 am, Kazuya FUKAMACHI wrote: 5) Toby's proposal I hope +1. I'm not satisfied with (1)-(4). So, I would like to wait for Toby's implementation. Ive not had as much time for this as I hoped, so please excuse any rough edges in these patch. All comments gratefully accepted. http://collector.zope.org/Zope/737 -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Fri, 20 Dec 2002 11:16:38 + Toby Dickenson [EMAIL PROTECTED] wrote: Ive not had as much time for this as I hoped, so please excuse any rough edges in these patch. All comments gratefully accepted. http://collector.zope.org/Zope/737 I've looked through each patches. I truly appreciate your time and efforts. I will call for Japanese Zope users to test those patches in their environment. Probably, they will accept and feel grateful, if all of them are included in Zope 2.6.1. I will report you later. Many thanks again. With kind regards Kazuya Fukamachi ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Friday 20 December 2002 2:22 pm, Kazuya FUKAMACHI wrote: if all of them are included in Zope 2.6.1. The final beta is due today. If that goes to plan then it is already too late. 2.6.2 is a realistic target. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Toby Dickenson [EMAIL PROTECTED] wrote: The final beta is due today. Was this announced somewhere? I really wish the release schedule was a bit more known in advance. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Fri, 20 Dec 2002 14:35:15 + Toby Dickenson [EMAIL PROTECTED] wrote: The final beta is due today. If that goes to plan then it is already too late. 2.6.2 is a realistic target. Ouch! Anyway, I will test them. Thanks Kazuya Fukamachi ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Toby Dickenson [EMAIL PROTECTED] wrote: The final beta is due today. Was this announced somewhere? I really wish the release schedule was a bit more known in advance. It was announced on this list (and zope-coders) a couple of weeks ago. FYI - with the holidays and other things going on, I don't think b2 will actually be able to go out until at least Monday. It looks like there might be a few left-over issues from bug day that I'd to follow up on, but I won't be available to do that today :( Brian Lloyd[EMAIL PROTECTED] V.P. Engineering 540.361.1716 Zope Corporation http://www.zope.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Ok sorry, I stand corrected. (BTW Brian your clock seems wrong) On Fri, 2002-12-20 at 03:52, Brian Lloyd wrote: Toby Dickenson [EMAIL PROTECTED] wrote: The final beta is due today. Was this announced somewhere? I really wish the release schedule was a bit more known in advance. It was announced on this list (and zope-coders) a couple of weeks ago. FYI - with the holidays and other things going on, I don't think b2 will actually be able to go out until at least Monday. It looks like there might be a few left-over issues from bug day that I'd to follow up on, but I won't be available to do that today :( -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi.
2.6.2 is a realistic target.
Sure.
I tried to your patch 03-properties.diff, then I found two problems.
1. browsers autodetection make a mistake.
dtml-var u' ' makes some web browsers confuse, it ignores meta tag
charset and autodetect UTF8.So I will remove dtml-var u' ', when
REQUEST has another charset value.
2. management_page_charset_tag really needs?
Because management_page_charset is approach of not using unicode.
But management_page_charset_tag exists, REQUEST will convert strings to
unicode strings.field2string has not convert_unicode method, then raise
unicodeerror. It is contradiction.
HTTPRequest.py line:499
if flagsCONVERTED:
try:
if character_encoding:
# We have a string with a specified character encoding.
# This gets passed to the converter either as unicode, if it can
# handle it, or crunched back down to latin-1 if it can not.
item = unicode(item,character_encoding)
if hasattr(converter,'convert_unicode'):
item = converter.convert_unicode(item)
else:
item = converter(item.encode('latin-1'))
I made a patch for fix those problems. It looks good work on my zope.
Regards.
--
Yusei TaharaSo it goes
[EMAIL PROTECTED]
properties.diff
Description: Binary data
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Fri, 20 Dec 2002 11:16:38 + Toby Dickenson [EMAIL PROTECTED] wrote: Ive not had as much time for this as I hoped, so please excuse any rough edges in these patch. All comments gratefully accepted. http://collector.zope.org/Zope/737 Wow! I'll check it out soon. Yusei already found some problems and made a fix, so I'll try his patch too. Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tue, 17 Dec 2002 12:32:11 +0900 Kazuya FUKAMACHI [EMAIL PROTECTED] wrote: I'm not sure which is better; 1) adopting new environment value, Z_SOMETHING ie. Z_DEFUALT_CHARSET ? 2) using locale.getlocale()[1] needs some abstruction layer umm, let's try some locale configuration: --- script start - from locale import setlocale, getlocale, LC_ALL import sys print sys.platform for l in ( 'en_US', 'en_US.ASCII', 'en_US.US-ASCII', 'en_US.latin-1', 'en_US.ISO-8859-1', 'en_US.ISO-8859-2', 'en_US.UTF-8', 'el_GR.ISO-8859-1', 'el_GR.ISO-8859-2', 'el_GR.ISO-8859-5', 'el_GR.ISO-8859-7', 'el_GR.ISO-2022', 'el_GR.EUC-JP', 'ja_JP.UTF-8', 'ja_JP.EUC-JP', 'ja_JP.ISO-2022', 'ja_JP.ISO-2022-JP', 'ja_JP.ISO-2022-JP-1', 'ja_JP.AJEC', 'ja_JP.PCK', 'ja_JP.ujis', 'ja_JP.Shift_JIS', 'ja_JP.x-sjis', 'ja_JP.SJIS', 'ja_JP.sjis', ): setlocale(LC_ALL, l) print '%-24s'% l, getlocale() --- script end - result on RHL6.2(En) --- result - linux2 en_US['en_US', 'ISO8859-1'] en_US.ASCII ['en_US', 'ISO8859-1'] en_US.US-ASCII ['en_US', 'ISO8859-1'] en_US.latin-1['en_US', 'ISO8859-1'] en_US.ISO-8859-1 ['en_US', 'ISO8859-1'] en_US.ISO-8859-2 ['en_US', 'ISO8859-1'] en_US.UTF-8 ['en_US', 'ISO8859-1'] el_GR.ISO-8859-1 ['el_GR', 'ISO8859-7'] el_GR.ISO-8859-2 ['el_GR', 'ISO8859-7'] el_GR.ISO-8859-5 ['el_GR', 'ISO8859-7'] el_GR.ISO-8859-7 ['el_GR', 'ISO8859-7'] el_GR.ISO-2022 ['el_GR', 'ISO8859-7'] el_GR.EUC-JP ['el_GR', 'ISO8859-7'] ja_JP.UTF-8 ['ja_JP', 'eucJP'] ja_JP.EUC-JP ['ja_JP', 'eucJP'] ja_JP.ISO-2022 ['ja_JP', 'eucJP'] ja_JP.ISO-2022-JP['ja_JP', 'eucJP'] ja_JP.ISO-2022-JP-1 ['ja_JP', 'eucJP'] ja_JP.AJEC ['ja_JP', 'eucJP'] ja_JP.PCK['ja_JP', 'eucJP'] ja_JP.ujis ['ja_JP', 'eucJP'] ja_JP.Shift_JIS ['ja_JP', 'eucJP'] ja_JP.x-sjis ['ja_JP', 'eucJP'] ja_JP.SJIS ['ja_JP', 'eucJP'] ja_JP.sjis ['ja_JP', 'eucJP'] --- result end - Wow, maybe we could say something like that?: 1) adopting new environment value - platform-neutral: no different behaviour on different platform - could set an encoding which mismatches with C's locale behaviour 2) using locale.getlocale()[1] - better accordance of locale setting and encoding setting - may affect on the behaviour of other components which depends on C's locale - the supported encodings are limited to the available ones on the platform - eliminate irregular combination of locale(lang+territory) + encoding Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Fri, 13 Dec 2002 11:12:22 +0900
Kazuya FUKAMACHI [EMAIL PROTECTED] wrote:
But, if it takes a few months or longer,
I will try to modify Russian patch as an experiment,
maybe taking in your approach in some degree.
Is that bad idea?
I believe it's good idea, as Russian patch's approach is a solid,
straightforward and standard way of I18N. Although not
good for M17N, the behaviour is simple straight so that it's
much less confusing for end-users (IMHO).
Probably os.environ.get('Z_SOMETHING') might be a better way than
locale.getlocale()[1], because using locale.getlocale()[1] means that
the behaviour of Zope will be changed implicitly, whereas
os.environ.get('Z_SOMETHING') is more explicit for the users, thus
less confusing..
Regards,
---
Heiichiro NAKAMURA [EMAIL PROTECTED]
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi.
Probably os.environ.get('Z_SOMETHING') might be a better way than
locale.getlocale()[1], because using locale.getlocale()[1] means that
the behaviour of Zope will be changed implicitly, whereas
os.environ.get('Z_SOMETHING') is more explicit for the users, thus
less confusing..
Nice idea.
We can get environment value everywhere in Zope.
it will be easy to make patch:-)
+1
Regards
--
Yusei
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tue, 17 Dec 2002 11:08:19 +0900
Yusei Tahara [EMAIL PROTECTED] wrote:
NAKAMURA wrote:
Probably os.environ.get('Z_SOMETHING') might be a better way than
locale.getlocale()[1], because using locale.getlocale()[1] means that
the behaviour of Zope will be changed implicitly, whereas
os.environ.get('Z_SOMETHING') is more explicit for the users, thus
less confusing..
Nice idea.
We can get environment value everywhere in Zope.
it will be easy to make patch:-)
+1
Zope 2.6.x has LOCALE_ID in 'z2.py' to set locale value.
If LOCALE_ID is set to something, locale.getlocale()[1] is sure
to get this value, because locale.setlocale(locale.LC_ALL, val)
is called from 'z2.py'.
For this reason, if you set LOCALE_ID, locale.getlocale()[1] does not
mean that behaviour of Zope will be changed implicitly, but rather
explicitly, I think.
I'm not sure which is better;
1) adopting new environment value, Z_SOMETHING
ie. Z_DEFUALT_CHARSET ?
2) using locale.getlocal()[1]
needs some abstruction layer
def getDefaultPythonCharEncodingName():
if os.name == 'posix':
return charEncodingMap.get(locale.getlocale()[1], 'latin1')
else: # For MS Windows
return os.environ.get('Z_CHAR_ENCODING', 'latin1')
[snip]
Any comments are welcomed.
Regards,
Kazuya Fukamachi
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Thank you for the detailed comment..
On Wed, 11 Dec 2002 11:15:37 +0900
Kazuya FUKAMACHI [EMAIL PROTECTED] wrote:
---
2) Russian patch:
http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html
+0.5
i) I like such an approach.
- select name=dtml-var id:utf8:text
+ select name=dtml-var id:dtml-var management_page_charset:text
i') using newly implemented function management_page_charset_default(),
it can set the value of default management_page_charset.
This avoids hard coding of default value.
ii) but this patch may have a few troubles in Japnanese environment.
This code returns 'eucJP' in many Japanese environment.
charset = locale.getlocale()[1]
And
codecs.lookup(charset) == codecs.lookup('eucJP')
will fail, because there are no entry for 'eucJP', but 'euc-jp'
and 'ujis'. I think it is possible to add 'eucJP' entry to
JapaneseCodecs as an alias for 'euc-jp'. So, it's not a big problem.
(I don't know why JapaneseCodecs doesn't have 'eucJP' alias.)
If the problem above has been solved,
the value of management_page_charset maybe set to 'eucJP',
and it leads to another problem.
If management_page_charset returns 'eucJP', then header should be
RESPONSE.setHeader('Content-Type','text/html; charset=eucJP')
It is not common way as a Content-Type header.
We prefer
RESPONSE.setHeader('Content-Type','text/html; charset=EUC-JP')
And also, it does not work in Windows environment.
This code returns (None, None).
locale.getlocale()[1]
I guess the problem is the difference of char-encoding naming
convention: even among Posix-complient OSes, the naming of encodings
are vender dependent (the situation is the same among RDBMS vendors).
If I were to use Russian patch, I might put one abstraction in the
char-encoding-name handling by providing some facilities like:
def getDefaultPythonCharEncodingName():
if os.name == 'posix':
return charEncodingMap.get(locale.getlocale()[1], 'latin1')
else: # For MS Windows
return os.environ.get('Z_CHAR_ENCODING', 'latin1')
def mapToIANA(encodingName):
get IANA encoding name for HTTP Header
return IANACharEncodingMap.get(encodingName, encodingName)
charEncodingMap = {
'PCK': 'Shift_JIS'
...
}
IANACharEncodingMap = {
'SJIS': 'Shift_JIS'
...
}
Sooner or later, I think this kind of mechanism will be required
for the mature support of Unicode, as Unicode brings a lot of
this kind of problems. Without the rational addressing of such issues,
the support of Unicode shouldn't be called mature I think.
Still I don't like this patch's approach very much because
this is the per-server-instance configuration, not useful
for building M17N web site.
iii) I guess modification to class PropertyManager seems
to fix http://collector.zope.org/Zope/697
Basically, it's interesting approach, but still needed to be brush up.
5) Toby's proposal
I hope +1.
I'm not satisfied with (1)-(4).
So, I would like to wait for Toby's implementation.
Probably it's a preferable choice.
My concern is I'm afraid if Toby is too busy to do that.
Since none of the choices(1-5) provide the perfect solution,
all of them are just a temporary patch for the urgent fix of
the severe issue (Collector 623).
So, I think it shouldn't take too much time (we shouldn't spend
too much time).
(further comments welcomed)
Regards,
---
Heiichiro NAKAMURA [EMAIL PROTECTED]
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
The current status of each resolutions (my own perception): --- 1) Nakagami Patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018177.html - security concern 2) Russian patch: http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html - some small modification should be done. 3) TAHARA's patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html - need fix of some problems 4) Rollback approach: Rollback some hassle code back to old safe one. - safer option(?) 5) Toby's proposal - relatively better consensus - not feasible if Toby is too busy to do that quickly. --- (Any comments/corrections welcomed) Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thu, 12 Dec 2002 15:51:48 -0800
Heiichiro NAKAMURA [EMAIL PROTECTED] wrote:
I guess the problem is the difference of char-encoding naming
convention: even among Posix-complient OSes, the naming of encodings
are vender dependent (the situation is the same among RDBMS vendors).
If I were to use Russian patch, I might put one abstraction in the
char-encoding-name handling by providing some facilities like:
Probably you're right.
I do agree your approach bellow.
def getDefaultPythonCharEncodingName():
if os.name == 'posix':
return charEncodingMap.get(locale.getlocale()[1], 'latin1')
else: # For MS Windows
return os.environ.get('Z_CHAR_ENCODING', 'latin1')
[snip]
IANACharEncodingMap = {
'SJIS': 'Shift_JIS'
...
}
Sooner or later, I think this kind of mechanism will be required
for the mature support of Unicode, as Unicode brings a lot of
this kind of problems. Without the rational addressing of such issues,
the support of Unicode shouldn't be called mature I think.
Still I don't like this patch's approach very much because
this is the per-server-instance configuration, not useful
for building M17N web site.
- per-server-instance configuration
- not useful for building M17N web site
I agree.
5) Toby's proposal
Probably it's a preferable choice.
My concern is I'm afraid if Toby is too busy to do that.
Yes, I'm also afraid so.
It's very difficult to provide a perfect solution.
Since none of the choices(1-5) provide the perfect solution,
all of them are just a temporary patch for the urgent fix of
the severe issue (Collector 623).
As a temporary patch,
+1
3) TAHARA's patch:
http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html
- need fix of some problems
It has some problems but easy to patch and start to use,
and also easy to remove after complete solution is provided.
If Toby will take some time, I will use 3) for a while.
But, if it takes a few months or longer,
I will try to modify Russian patch as an experiment,
maybe taking in your approach in some degree.
Is that bad idea?
So, I think it shouldn't take too much time (we shouldn't spend
too much time).
Maybe, we should separate the problem into temporary patch
and complete solution, if it takes much time.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tuesday 10 December 2002 12:03 am, Kazuya FUKAMACHI wrote: On Mon, 09 Dec 2002 13:18:26 -0800 Heiichiro NAKAMURA [EMAIL PROTECTED] wrote: 2. Whenever any experimental enhancements to the ZMI which rely on using Unicode is to be integrated, create an new tab and put these features in that page in order: Sounds reasonable. USA is #1 in Internet Users with 160M. http://www.etforecasts.com/pr/pr1202.htm At least 21.91% of Internet users are CJK users. (Japan + China + South Korea + Taiwan) Since using UTF-8 for their pages is still not in common in these area, it is preferable to accomodate such workaround. I dont follow the logic here. Using unicode in the server doesnt force you to use utf8 in the browser. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Monday 09 December 2002 9:18 pm, Heiichiro NAKAMURA wrote: On Mon, 9 Dec 2002 10:13:16 + I agree with everything, except This is a tentative limitation on UI design, and should be removed after the Unicode Support gets matured enough to handle Unicode object with all languages/encodings. Thats not right. The core unicode support has been mature in this area for over a year. The incompatability with pre-encoded strings is a design choice - not something that will improve with time. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Now there are five candidates of resolution for the Collector 623 issue raised: --- 1) Nakagami Patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018177.html 2) Russian patch: http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html 3) TAHARA's patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html 4) Rollback approach: Rollback some hassle code back to old safe one. 5) Toby's proposal --- Probably we should go to the next step:evaluation, decision, implementation. When can we implement one of them and apply to the official release? Time required for implementation (IMHO): (1)-(3) are patches which are already written. (4) is disabling some features, not very hard. (5) is a little tricky (not hard though), not sure how long it takes. With which official version is this issue resolved? 1. As a hotfix of Ver. 2.6.0 2. Ver. 2.6.1 3. As a hotfix of Ver. 2.6.1 4. Ver. 2.6.2 5. As a hotfix of Ver. 2.6.2 My impression is: - In terms of implementation time, (1)-(4) look safer. - In terms of consensus, (5) seems safer. - In terms of robustness, (4) looks safer, as it doesn't add newly written code almost at all. Any opinion from anyone is welcomed. Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tue, 10 Dec 2002 12:12:19 -0800
Heiichiro NAKAMURA [EMAIL PROTECTED] wrote:
---
2) Russian patch:
http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html
+0.5
i) I like such an approach.
- select name=dtml-var id:utf8:text
+ select name=dtml-var id:dtml-var management_page_charset:text
i') using newly implemented function management_page_charset_default(),
it can set the value of default management_page_charset.
This avoids hard coding of default value.
ii) but this patch may have a few troubles in Japnanese environment.
This code returns 'eucJP' in many Japanese environment.
charset = locale.getlocale()[1]
And
codecs.lookup(charset) == codecs.lookup('eucJP')
will fail, because there are no entry for 'eucJP', but 'euc-jp'
and 'ujis'. I think it is possible to add 'eucJP' entry to
JapaneseCodecs as an alias for 'euc-jp'. So, it's not a big problem.
(I don't know why JapaneseCodecs doesn't have 'eucJP' alias.)
If the problem above has been solved,
the value of management_page_charset maybe set to 'eucJP',
and it leads to another problem.
If management_page_charset returns 'eucJP', then header should be
RESPONSE.setHeader('Content-Type','text/html; charset=eucJP')
It is not common way as a Content-Type header.
We prefer
RESPONSE.setHeader('Content-Type','text/html; charset=EUC-JP')
And also, it does not work in Windows environment.
This code returns (None, None).
locale.getlocale()[1]
iii) I guess modification to class PropertyManager seems
to fix http://collector.zope.org/Zope/697
Basically, it's interesting approach, but still needed to be brush up.
5) Toby's proposal
I hope +1.
I'm not satisfied with (1)-(4).
So, I would like to wait for Toby's implementation.
1) Nakagami Patch:
http://lists.zope.org/pipermail/zope-dev/2002-November/018177.html
As I wrote sometime ago, this approach has problems.
3) TAHARA's patch:
http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html
He said;
Fri, 06 Dec 2002 11:30:39 +0900
Sorry, I noticed that my patch is defective.
It is failure when object has own Property:-(
4) Rollback approach:
Rollback some hassle code back to old safe one.
This is the last resort.
Regards,
Kazuya Fukamachi
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tue, 10 Dec 2002 12:52:06 -0800
Heiichiro NAKAMURA [EMAIL PROTECTED] wrote:
On Tue, 10 Dec 2002 08:52:09 +
Toby Dickenson [EMAIL PROTECTED] wrote:
On Tuesday 10 December 2002 12:03 am, Kazuya FUKAMACHI wrote:
On Mon, 09 Dec 2002 13:18:26 -0800
http://www.etforecasts.com/pr/pr1202.htm
At least 21.91% of Internet users are CJK users.
(Japan + China + South Korea + Taiwan)
Since using UTF-8 for their pages is still not in common
in these area, it is preferable to accomodate such workaround.
I dont follow the logic here. Using unicode in the server doesnt force you to
use utf8 in the browser.
As far as Toby's approach does not force me to use utf8 on ZMI,
using unicode internally might not be a big problem, maybe.
I say 'maybe', because I still cannot correctly grasp how far
new features' unicode objects will make an effect on legacy code.
So, I just worrying about.
Thank you, Heiichiro.
What you wrote 1)-4) is nearly what I think.
Regardig 5)-7), it's not exactly same with my opinion.
I have an ambivalent opinion about using of Unicode.
But, for the moment, it's not essential issue here,
I'm not going to discuss now. Thanks, anyway.
5) Therefore, they have to use 8bit-string object in order to
use their language text in Zope.
6) If we could use UTF-8 without trouble in any situation, things
are much easier since you can assume anything in Unicode like:
input name=var1:utf8:ustring,
REQUEST.set('management_page_charset','UTF-8'),
RESPONSE.setHeader('Content-Type','text/html; charset=UTF-8'),
Text is handled as Unicode Object in Zope internal,
But it's just a fantasy that I18N engineers often stick to.
7) Overall, the only practical way for them is not to use Unicode
in Zope and deal with text as raw 8bit string.
Regards,
Kazuya Fukamachi
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Monday 09 December 2002 2:20 am, Heiichiro NAKAMURA wrote:
On Sun, 8 Dec 2002 21:58:16 +
Toby Dickenson [EMAIL PROTECTED] wrote:
On Thursday 05 December 2002 9:36 pm, Toby Dickenson wrote:
Yes, I have an idea. I hope to find time to flesh it out early next
week.
I propose:
Here is my understanding of your proposal:
1. management_page_charset property for non-Latin-1
Affected application: ZMI only
Limitation:
- Standard Objects in Zope (such as title property) must be
Non-Unicode.
Yes, if using management_page_charset.
- All objects must be Non-Unicode in order to work in ZMI.
To be precise, all objects rendered on the management page must be plain 8bit
strings.
- If there is even one Unicode object there, all data will be
assumed as 'Latin-1' and non-Latin-1 data cannot be used in
any object.
yes
- REQUEST.set('management_page_charset','UTF-8') does not work
when 'management_page_charset' is set as global property.
That is the current, broken behaviour. I propose that REQUEST.set overrides
the global property. Note that no standard ZMI page will do so, except the
'properties' tab explained below.
Homework (new issues):
- define the safe steps of implementation of Unicode Support
- find a way of smooth migration of MBCS-to-Unicode
Yes. There is nothing zope-specific about this being hard ;-)
2. Behaviour of handling of text in ZMI
Not all of ZMI - this is just the 'Properties' tab.
When UnicodeType Properties are contained:
- internal representation: 'UCS-2/UTF-16' if UnicodeType(ex. ustring)
'Latin-1' if not UnicodeType(ex. string)
- encoding of output data: 'Latin-1'
No, output over http will be utf8.
- encoding of input data: 'Latin-1'
If you mean input from the browser when a property changes, no. it will use
utf8.
When UnicodeType Properties are NOT contained:
- encoding of input data: any
yes, the same as all other ZMI pages. latin-1 by default, and can be overidden
by management_page_charset.
- internal representation: same as input (no conversion)
- encoding of output data: same as internal (no conversion)
Limitation:
- Non-Latin-1 value cannot be used when creating the first Unicode
Property.
(Any correction is welcomed)
I'll ask power users of Japan-Zope-User-Group-ML about opinions
regarding to the proposal.
but I
suggest that compatability with this feature should not hold back any
future enhancements to the ZMI which rely on using unicode)
I guess this statement is somewhat ambiguous. Probably we could say
something like that? :
1. Any future enhancements to the ZMI which rely on using unicode
should be carefully defined, examined, evaluated and feasibility-tested
so that all issues/limitations can be clarified and the consequent
impact of these issues/limitation can be evaluated in order to
avoid hassle implementation integrated into the official version.
yes
2. Any experimental enhancements to the ZMI which rely on using unicode
must be integrated into the official version in the manner that
it doesn't affect the behaviour of 'legacy' applications,
I disagree. I dont want to block a good new feature from Zope 2.x (x=7)
simply because it uses unicode in a way that is incompatible with the
management_page_charset property. Thats what 'legacy' means.
until
the Unicode Support gets matured enough to handle Unicode object
with all languages/encodings.
Yes. There are several places in the ZMI that could benefit from
'unicodization'. At the moment I suspect it will be difficult to implement
these improvements while retaining support for management_page_charset.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Mon, 9 Dec 2002 10:13:16 +
Toby Dickenson [EMAIL PROTECTED] wrote:
2. Any experimental enhancements to the ZMI which rely on using unicode
must be integrated into the official version in the manner that
it doesn't affect the behaviour of 'legacy' applications,
I disagree. I dont want to block a good new feature from Zope 2.x (x=7)
simply because it uses unicode in a way that is incompatible with the
management_page_charset property. Thats what 'legacy' means.
I see. How about that then?
2. Whenever any experimental enhancements to the ZMI which rely on
using Unicode is to be integrated, create an new tab and put these
features in that page in order:
- to provide a choice of not to use new features so that
you can avoid serious problems which may be caused by them, thus
traditional implementations (ex. 'Contents', 'View', 'Properties'
tabs) works fine and their behaviours aren't affected by new features.
- to make users be able to distinguish which features are new-and-risky
easily, so that users can avoid using potentially hazardous
implementation more easily. Maybe it's better to change the bgcolor
of these tab and put the warning message on top of the new pages
of these features.
This is a tentative limitation on UI design, and should be removed
after the Unicode Support gets matured enough to handle Unicode object
with all languages/encodings.
My intension is:
- Not to block the integration of new features which use Unicode
- Not to make Zope un-usable for Non-Latin-1-language-native users:
- At least the essential features should work for anyone.
- It's OK if the new nice-to-have features do not work.
I revised some parts of my note based on your input. Here's the new one:
1. management_page_charset property for non-Latin-1 in ZMI
Purpose:
- Use management_page_charset property to use Non-Latin-1
text as Non-Unicode (plain-8bit-string) text in ZMI.
Affected application: ZMI only
Limitation:
- Standard Objects in Zope (such as title property) must be
Non-Unicode (if using management_page_charset).
- All objects in the management page must be plain-8bit-string
in order to work in ZMI.
- If there is even one Unicode object there, all data will be
assumed as 'Latin-1' and non-Latin-1 data cannot be used in
any object.
- In the current release (original 2.6.0 release)
REQUEST.set('management_page_charset','UTF-8') does not work
when 'management_page_charset' is set as global property.
This limitation will be fixed: the value of REQUEST.set
overrides the global property (in 'properties' page only).
Homework (new issues):
- define the safe steps of implementation of Unicode Support
- find a way of smooth migration of MBCS-to-Unicode
2. Behaviour of handling of text in ZMI's 'Properties' page.
When UnicodeType Properties are contained:
- encoding of HTTP input data: 'UTF-8'
- internal representation: 'UCS-2/UTF-16' if UnicodeType(ex. ustring)
'Latin-1' if not UnicodeType(ex. string)
- encoding of HTTP output data: 'UTF-8'
When UnicodeType Properties are NOT contained:
- encoding of HTTP input data: any
(default:Latin-1, can be overridden by 'management_page_charset')
- internal representation: same as input (no conversion)
- encoding of HTTP output data: same as internal (no conversion)
Limitation:
- Non-Latin-1 value cannot be used when creating the first Unicode
Property.
3. Guideline on the future enhancements to the ZMI which rely on using Unicode
3.1. Any future enhancements to the ZMI which rely on using Unicode
should be carefully defined, examined, evaluated and feasibility-tested
so that all issues/limitations can be clarified and the consequent
impact of these issues/limitations can be evaluated in order to
avoid hassle implementation integrated into the official version.
3.2. Whenever any experimental enhancement to the ZMI which rely on
using Unicode is to be integrated, create new tabs and put these
features in these pages in order:
- to provide a choice of not to use new features so that
you can avoid serious problems which may be caused by them, thus
traditional implementations (ex. 'Contents', 'View', 'Properties'
tabs) works fine and their behaviours aren't affected by new features.
- to make users be able to distinguish which features are
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Mon, 09 Dec 2002 13:18:26 -0800 Heiichiro NAKAMURA [EMAIL PROTECTED] wrote: 2. Whenever any experimental enhancements to the ZMI which rely on using Unicode is to be integrated, create an new tab and put these features in that page in order: Sounds reasonable. USA is #1 in Internet Users with 160M. http://www.etforecasts.com/pr/pr1202.htm At least 21.91% of Internet users are CJK users. (Japan + China + South Korea + Taiwan) Since using UTF-8 for their pages is still not in common in these area, it is preferable to accomodate such workaround. --- Kazuya Fukamachi ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thursday 05 December 2002 9:36 pm, Toby Dickenson wrote:
On Thursday 05 December 2002 8:41 pm, Heiichiro NAKAMURA wrote:
Does anyone have any other idea for the Collector 623 issue?
I hope better ideas will be posted..
Yes, I have an idea. I hope to find time to flesh it out early next week.
I propose:
1. Currently you can set a management_page_charset property to override the
global assumption that legacy ZMI pages are latin1. This is a hack that
happens to mostly work by accident, provided your ZMI pages do not encounter
a unicode object. I propose promoting this to a standard documented feature.
Is there anyone who actually uses this feature who can contribute some
documentation?
(Note that feature will never work with pages that do encounter unicode
objects. This means Zope should try to avoid gratuitous unicode usage, but I
suggest that compatability with this feature should not hold back any future
enhancements to the ZMI which rely on using unicode)
2. The documented way of setting a ZMI-page-specific encoding is
REQUEST.set('management_page_charset','UTF-8'). Currently
management_page_charset as a global property will override this page specific
setting. This is a bug that needs to be fixed.
3. I propose changing the encoding used by the standard 'Properties' ZMI page.
Currently it uses UTF8 always, and assumes that 8bit string properties are
latin1. I propose that this policy is used only if a unicode property has
already been defined on this object. If it has not, it uses the same encoding
policy as every other ZMI page - that is the value of the
management_page_charset property, or latin-1 if it has not been set.
A disadvantage of this change is that it will not be possible to set a
non-latin-1 initial value when creating the first unicode property. I think
this is just about acceptable.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Sun, 8 Dec 2002 21:58:16 +
Toby Dickenson [EMAIL PROTECTED] wrote:
On Thursday 05 December 2002 9:36 pm, Toby Dickenson wrote:
Yes, I have an idea. I hope to find time to flesh it out early next week.
I propose:
Here is my understanding of your proposal:
1. management_page_charset property for non-Latin-1
Affected application: ZMI only
Limitation:
- Standard Objects in Zope (such as title property) must be
Non-Unicode.
- All objects must be Non-Unicode in order to work in ZMI.
- If there is even one Unicode object there, all data will be
assumed as 'Latin-1' and non-Latin-1 data cannot be used in
any object.
- REQUEST.set('management_page_charset','UTF-8') does not work
when 'management_page_charset' is set as global property.
Homework (new issues):
- define the safe steps of implementation of Unicode Support
- find a way of smooth migration of MBCS-to-Unicode
2. Behaviour of handling of text in ZMI
When UnicodeType Properties are contained:
- encoding of input data: 'Latin-1'
- internal representation: 'UCS-2/UTF-16' if UnicodeType(ex. ustring)
'Latin-1' if not UnicodeType(ex. string)
- encoding of output data: 'Latin-1'
When UnicodeType Properties are NOT contained:
- encoding of input data: any
- internal representation: same as input (no conversion)
- encoding of output data: same as internal (no conversion)
Limitation:
- Non-Latin-1 value cannot be used when creating the first Unicode
Property.
(Any correction is welcomed)
I'll ask power users of Japan-Zope-User-Group-ML about opinions
regarding to the proposal.
but I
suggest that compatability with this feature should not hold back any
future enhancements to the ZMI which rely on using unicode)
I guess this statement is somewhat ambiguous. Probably we could say
something like that? :
1. Any future enhancements to the ZMI which rely on using unicode
should be carefully defined, examined, evaluated and feasibility-tested
so that all issues/limitations can be clarified and the consequent
impact of these issues/limitation can be evaluated in order to
avoid hassle implementation integrated into the official version.
2. Any experimental enhancements to the ZMI which rely on using unicode
must be integrated into the official version in the manner that
it doesn't affect the behaviour of 'legacy' applications, until
the Unicode Support gets matured enough to handle Unicode object
with all languages/encodings.
---
Heiichiro NAKAMURA [EMAIL PROTECTED]
On Sun, 8 Dec 2002 21:58:16 +
Toby Dickenson [EMAIL PROTECTED] wrote:
On Thursday 05 December 2002 9:36 pm, Toby Dickenson wrote:
On Thursday 05 December 2002 8:41 pm, Heiichiro NAKAMURA wrote:
Does anyone have any other idea for the Collector 623 issue?
I hope better ideas will be posted..
Yes, I have an idea. I hope to find time to flesh it out early next week.
I propose:
1. Currently you can set a management_page_charset property to override the
global assumption that legacy ZMI pages are latin1. This is a hack that
happens to mostly work by accident, provided your ZMI pages do not encounter
a unicode object. I propose promoting this to a standard documented feature.
Is there anyone who actually uses this feature who can contribute some
documentation?
(Note that feature will never work with pages that do encounter unicode
objects. This means Zope should try to avoid gratuitous unicode usage, but I
suggest that compatability with this feature should not hold back any future
enhancements to the ZMI which rely on using unicode)
2. The documented way of setting a ZMI-page-specific encoding is
REQUEST.set('management_page_charset','UTF-8'). Currently
management_page_charset as a global property will override this page specific
setting. This is a bug that needs to be fixed.
3. I propose changing the encoding used by the standard 'Properties' ZMI page.
Currently it uses UTF8 always, and assumes that 8bit string properties are
latin1. I propose that this policy is used only if a unicode property has
already been defined on this object. If it has not, it uses the same encoding
policy as every other ZMI page - that is the value of the
management_page_charset property, or latin-1 if it has not been set.
A disadvantage of this change is that it will not be possible to set a
non-latin-1
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
So far, I believe there have been three choices of resolutions raised, for the Collector 623 issue: 1) Nakagami Patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018177.html 2) Russian patch: http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html 3) TAHARA's patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html Probably the forth choice could be: - Change the data-type of title property from usting to string - Prohibit the manipulation of utext/ustring/ulines/utokens properties from ZMI: To be specific, remove these from properties.dtml: optionutext/option optionutokens/option optionulines/option optionustring/option etc. --- PROS of the forth approach --- - Old safe way: old (2.5.1 base) products (including other languages) now should work fine. - Can avoid the risk of changing type of 'title' of standard Zope types such as 'Folder' into 'ustring' and causing consequent new problems - No new code is integrated: basically it's the rollback of some code to old (2.5.1) one, so safer solution in terms of wrong-specification risk, new-bug-introducing risk, etc. - Unicode Properties are still available except from ZMI. --- CONS of the forth approach --- - Need to modify new code which assumes title property is Unicode So, here's the list of resolution choices for the Collector 623 issue: --- 1) Nakagami Patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018177.html 2) Russian patch: http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html 3) TAHARA's patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html 4) Rollback approach: Rollback some hassle code back to old safe one. --- Does anyone have any other idea for the Collector 623 issue? I hope better ideas will be posted.. Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thursday 05 December 2002 8:41 pm, Heiichiro NAKAMURA wrote: Does anyone have any other idea for the Collector 623 issue? I hope better ideas will be posted.. Yes, I have an idea. I hope to find time to flesh it out early next week. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thu, 5 Dec 2002 21:36:49 + Toby Dickenson [EMAIL PROTECTED] wrote: Does anyone have any other idea for the Collector 623 issue? I hope better ideas will be posted.. Yes, I have an idea. I hope to find time to flesh it out early next week. Good news. Appreciated your time on this (everybody knows you are a busy person). Now we have five resolution choices for the Collector 623 issue: --- 1) Nakagami Patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018177.html 2) Russian patch: http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html 3) TAHARA's patch: http://lists.zope.org/pipermail/zope-dev/2002-November/018198.html 4) Rollback approach: Rollback some hassle code back to old safe one. 5) Toby's approach will be announced probably in next week --- Any other idea will be welcomed.. Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi. Sorry, I noticed that my patch is defective. It is failure when object has own Property:-( -- Yusei ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi. I made monkey patch for myself, when management_page_charset is not UTF-8, this patch remove :utf8: from non unicode type input field. because if input values are not latin-1, then unicode error raised. I think that every encoding can be used it satisfactory:-) Please consider whether it can merge into Zope2.6.1. There are some details missing from your explanation, but hopefully not from your patch. where do I find it? What thing is concretely some details? I'm very interested in Zope development, especially i18n. So I would like to contribute something about it:-) Thank you. -- Yusei TaharaSo it goes [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tuesday 03 December 2002 1:16 pm, Yusei TAHARA wrote: Hi. I made monkey patch for myself, when management_page_charset is not UTF-8, this patch remove :utf8: from non unicode type input field. because if input values are not latin-1, then unicode error raised. I think that every encoding can be used it satisfactory:-) Please consider whether it can merge into Zope2.6.1. There are some details missing from your explanation, but hopefully not from your patch. where do I find it? What thing is concretely some details? The fix to this situation is more complicated than removing a :utf8: from somewhere that it shouldnt be. Im sure you know this. I'm very interested in Zope development, especially i18n. So I would like to contribute something about it:-) The patch you mentioned? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi. What thing is concretely some details? The fix to this situation is more complicated than removing a :utf8: from somewhere that it shouldnt be. Im sure you know this. At this point Zope2.6.x, my guess is that would not clear up this problem completely.But my patch allows to use both type string and ustring when a user set right encoding name to management_page_charset. This is a temporary trick, but works well. I'm very interested in Zope development, especially i18n. So I would like to contribute something about it:-) The patch you mentioned? I want to tackle this complicated problem in Zope development rather than make my own patches. Because the i18n is more benefit for minority languages such as japanese than major one.I need it very much. Thank you. -- Yusei ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Tue, 3 Dec 2002 14:52:31 + Toby Dickenson [EMAIL PROTECTED] wrote: There are some details missing from your explanation, but hopefully not from your patch. where do I find it? What thing is concretely some details? The fix to this situation is more complicated than removing a :utf8: from somewhere that it shouldnt be. Im sure you know this. I'm very interested in Zope development, especially i18n. So I would like to contribute something about it:-) The patch you mentioned? ... seems like the things are in double bind situation, where the current Unicode Support has a fundamental problem which can't be resolved in straight manner, yet you are requesting it and rejecting any other proposal (at least looks so to some extent, even though you didn't intend to do so). # Probably one of the major cause of the miscommunication about # this topic is the difficulty of the understanding of different # culture, especially the sense of criticality and severity # of the problem: one party feels it's crucial and needs urgent # tentative fix while another feels it's minor and negligible. Although the problem itself is very clear, no actions/progress against the problem have happened yet. Anyway, I think it's getting clear that there are no silver bullets (patch) to shoot this problem. If so, the next step which should be taken in such a situation is clear: rollback. We should trace back until finding out what exactly is the fundamental cause of the problem, then make the corrections or define the right direction (as Toby already tried in [EMAIL PROTECTED] and I think it's comprehensive). I'm not clear the exact positioning of these documents: http://www.zope.org/Members/htrd/howto/unicode-zdg-changes http://www.zope.org/Members/htrd/howto/unicode but it seems like these documents describes the basic design policy of the current implementation of Unicode Support well. # For me, the problem looks much simpler in these documents rather # than the discussion about the detailed implementation. So far, does it sound OK? I would like to hear any opinion about this issue (I know anybody in this ML is very busy ;) Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thu, 28 Nov 2002 18:25:26 + Toby Dickenson [EMAIL PROTECTED] wrote: Feel free to do that if it works for you, but a patch for this approach will not be accepted into the standard Zope. See this bug report for an explanation of why this contradict's Zope's current unicode approach: http://collector.zope.org/Zope/633 One question. I don't understand what do you mean Zope's current unicode approach very well. Do you mean http://www.zope.org/Members/htrd/howto/unicode-zdg-changes http://www.zope.org/Members/htrd/howto/unicode are the Zope's current unicode approach? Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Saturday 30 November 2002 5:39 am, Yusei TAHARA wrote: 2. ustring can not join or replace to 8-bit strings other than ascii. Yes, it is painful to work in a mix of pre-encoded 8 bit strings and unicode strings. Thats why Zope's unicode policy was intended to be entirely optional - at least in the short term - and provided you can avoid unicode objects entirely. Zope's approach is that your application (excluding the ZMI for now) should work as in 2.5, provided you dont use any unicode values. Can you confirm that this is working OK? yes. If I input values by PythonScript without using property tab, it works well. Hurrah. One area where this went wrong for you is the properties page, which is explicitly set to utf8 to allow unicode properties to be set. Im not yet sure on the best way to fix this for you, but I agree it needs fixing. I made monkey patch for myself, when management_page_charset is not UTF-8, this patch remove :utf8: from non unicode type input field. because if input values are not latin-1, then unicode error raised. I think that every encoding can be used it satisfactory:-) Please consider whether it can merge into Zope2.6.1. There are some details missing from your explanation, but hopefully not from your patch. where do I find it? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Friday 29 November 2002 5:44 am, Yusei Tahara wrote:
Hi.
The right approach is to make it possible to change the title property to
a unicode string. All my custom products have this already, but it is a
deficiency in the standard Zope types such as 'Folder' that their titles
type can not be changed.
This approach is not useful for me.
I often use zope with RDB like MySQL.
generally japanese encoding text is in RDB.
so if object properties are unicode string,
I always encode it before publishing.
title tal:content=python:here.title.encode('euc-jp')TITLE/title
because, it always mix in RDB data(japanese) and
properties(python unicode string) in one page.
certainly I will be faced with this ustring problem everytime.
Japanese charactor is not ascii,
so if I do not encode ustring before publishing,raise unicode error.
The unicode changes were designed to cope with your scenario, so I would like
to explore the limits of where this approach has gone wrong.
As I understand it, you dont want to do anything in unicode. You store,
manipulate, process, and output strings as 8-bit pre-encoded strings. You
are making an assumption that these 8-bit strings use some specific character
encoding. The scope of this assumption is quite broad. It has to cover:
1. All strings stored by your ZODB
2. All strings stored by your RDB.
3. All processing performed by Zope products. (must follow your assumptions,
or be encoding neutral)
Anything that beaks these assumptions will need special treatment.
Am I right so far?
Im sure you are aware of the limitations of this approach - they are the
limitations that unicode was designed to break (not Zope's unicode, but
unicode in general). Im not sure your approach is workable long term because
an increasing number of Zope products will naturally use unicode so that they
can store properties containing text across a range of scripts. Even if this
is not workable long term, it was supposed to work in 2.6.
Zope's approach is that your application (excluding the ZMI for now) should
work as in 2.5, provided you dont use any unicode values. Can you confirm
that this is working OK?
Zope 2.5 left ZMI character encoding down to browser autodetection, and as a
result most ZMI-controlled properties are encoding-neutral. For compatability
with unicode pages, which are seen as the long term future, the ZMI has to
specify *some* character encoding. By default in 2.6 this is latin-1, a
change which I think was announced mid-way through the 2.5 development cycle.
I understand that some users are very happy overriding this with a
management_page_charset property on their root folder. Ive never used this,
and it wasnt designed to work this way, but it looks like it works due to
happy coincidence. Note that this doesnt work for management pages which
explicitly set their own character encoding, or management pages which
'accidentally' encounter a unicode string.
(Although this management_page_charset feature was not planned, I guess I will
be supporting this feature if it is being used)
One area where this went wrong for you is the properties page, which is
explicitly set to utf8 to allow unicode properties to be set. Im not yet sure
on the best way to fix this for you, but I agree it needs fixing.
As far as I can tell this is the only area which should be causing you
problems. Am I missing something?
(The 'title' problem I mentioned in a previous post is also a problem that
needs fixing, but not one that affects you if you dont want to use unicode.)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi. As I understand it, you dont want to do anything in unicode. You store, manipulate, process, and output strings as 8-bit pre-encoded strings. You are making an assumption that these 8-bit strings use some specific character encoding. The scope of this assumption is quite broad. It has to cover: 1. All strings stored by your ZODB 2. All strings stored by your RDB. 3. All processing performed by Zope products. (must follow your assumptions, or be encoding neutral) Anything that beaks these assumptions will need special treatment. Am I right so far? yes. why I don't want to use ustring, because there are two problems for me. 1. the original encoding is not clear anymore. It does not know that it will change at once. there is no encoding which raise unicode error other than utf-8. but utf-8 is not common encoding in japanese. almost all PDAs or cellular phones are not supporting utf-8. Probably, automatic encoding detection will be required in order to become practical. 2. ustring can not join or replace to 8-bit strings other than ascii. I have an same experience in Plone i18n with Localizer and TranslationService. Zope's approach is that your application (excluding the ZMI for now) should work as in 2.5, provided you dont use any unicode values. Can you confirm that this is working OK? yes. If I input values by PythonScript without using property tab, it works well. Zope 2.5 left ZMI character encoding down to browser autodetection, and as a result most ZMI-controlled properties are encoding-neutral. For compatability with unicode pages, which are seen as the long term future, the ZMI has to specify *some* character encoding. By default in 2.6 this is latin-1, a change which I think was announced mid-way through the 2.5 development cycle. I understand that some users are very happy overriding this with a management_page_charset property on their root folder. Ive never used this, and it wasnt designed to work this way, but it looks like it works due to happy coincidence. Note that this doesnt work for management pages which explicitly set their own character encoding, or management pages which 'accidentally' encounter a unicode string. I think management_page_charset is very useful, if it function correctly. One area where this went wrong for you is the properties page, which is explicitly set to utf8 to allow unicode properties to be set. Im not yet sure on the best way to fix this for you, but I agree it needs fixing. I made monkey patch for myself, when management_page_charset is not UTF-8, this patch remove :utf8: from non unicode type input field. because if input values are not latin-1, then unicode error raised. I think that every encoding can be used it satisfactory:-) Please consider whether it can merge into Zope2.6.1. Thank you -- Yusei TaharaSo it goes [EMAIL PROTECTED] properties.dtml Description: Binary data
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thu, 28 Nov 2002 09:47:04 +0900 Hajime Nakagami [EMAIL PROTECTED] wrote: Zope 2.6.0 have a problem for non 'iso-8859-1' user. And Collector 623 is still incollect now. http://collector.zope.org/Zope/623 It's serious problem for CJK(I'm Japanese). So I make patch for 2.6.0. http://www005.upp.so-net.ne.jp/zope260-i18n-20021123.diff This patch seems to have a minor performance problem, that is, applying this patch, traverse() be called two times at the time of publishing. 1. ZPublisher/HTTPRequest.py - get_default_charset() - added to get default charset from the environment 2. ZPublisher/Publish.py - publish() - normal call I would like to know whether calling traverse() only to get 'default_charset' is inevitable. Any good workaround? Another thing, I'm concerning over security issues. In this patch, extra argument is added to traverse(); def traverse(self, path, response=None, validated_hook=None, auth_check=1): -- auth_check is added And called like this from get_default_charset(); object=req.traverse(req.environ['PATH_INFO'][:], auth_check=0) Putting auth_check=0 will bypass authorization check. I don't want to add such an argument, because it might bear a security issue. Any good workaround? I'm hesitating to upgrading to Zope 2.6.0, partly because of http://collector.zope.org/Zope/623. It would be greatly appreciated if this patch would be refined. Regards, Kazuya -- Kazuya Fukamachi The limits of my language are http://www.atransia.co.jp/home/ZenKai/ the limits of my world. (sorry only in Japanese) --Ludwig Wittgenstein ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
On Thu, 28 Nov 2002 18:25:26 + Toby Dickenson [EMAIL PROTECTED] wrote: Feel free to do that if it works for you, but a patch for this approach will not be accepted into the standard Zope. See this bug report for an explanation of why this contradict's Zope's current unicode approach: It seems some Russian made a similar patch to solve the same problem too: http://itconnection.ru/pipermail/zopyrus/2002-November/001388.html I hate to say it, but it seems the Zope's current unicode approach is very problematic and offensive to the Zope community outside Latin-1 world. As of late, in Japanese Zope community, 2.6.0 seems to be regarded as defective and almost all users skip it rather than upgrade from 2.5.1, as far as I know. I'm at a loss, as to what to do, what's the solution/future direction, etc. Regards, --- Heiichiro NAKAMURA [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Hi.
The right approach is to make it possible to change the title property to a
unicode string. All my custom products have this already, but it is a
deficiency in the standard Zope types such as 'Folder' that their
titles type can not be changed.
This approach is not useful for me.
I often use zope with RDB like MySQL.
generally japanese encoding text is in RDB.
so if object properties are unicode string,
I always encode it before publishing.
title tal:content=python:here.title.encode('euc-jp')TITLE/title
because, it always mix in RDB data(japanese) and
properties(python unicode string) in one page.
certainly I will be faced with this ustring problem everytime.
Japanese charactor is not ascii,
so if I do not encode ustring before publishing,raise unicode error.
if don't allow to use string type in properties, I regret that I
would continue to use zope2.5.1 or make my own monkey patches for
after this.
I'm sorry to my BAD English.
Thank you.
--
Yusei TAHARA
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope 2.6.0 ZMI Problem for CJK(Collector 623) patch.
Sorry http://www005.upp.so-net.ne.jp/nakagami/zope260-i18n-20021123.diff http://www005.upp.so-net.ne.jp/zope260-i18n-20021123.diff ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
