Re: Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
On 10/29/06, Yoshinori Okuji [EMAIL PROTECTED] wrote: IIRC, Jim noted that it would be better to backport zope3's, while our patches are for zope2's. See http://mail.zope.org/pipermail/zope-dev/2006-May/027503.html for a short thread on successfully using xmlpickle. With that recipe and patch I was able to import and export a Plone site at the time. Michael ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
On 10/28/06, Yoshinori Okuji [EMAIL PROTECTED] wrote: for now, so I don't think I can do that. If it is acceptable to simply fix the current code, we can provide patches. Of course it's acceptable. Although it would as previously mentioned be better if you could check them in yourselves instead. And if they could write some unittests that would be even better. :) Sure, it is a good thing to write tests. If you just provide patches, tests are necessary. Nobody likes to check in others code without clear tests to show what it does. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
On Friday 27 October 2006 16:46, Lennart Regebro wrote: I would propose that somone that uses this heavily (like the Nexedi people), joins the Zope Foundation as a commiter, gets commiter rights and holds his or her hands over this functionality. :) We are definitely willing to help the maintenance of this feature, but I am not sure what should be done. IIRC, Jim noted that it would be better to backport zope3's, while our patches are for zope2's. I don't have much time for now, so I don't think I can do that. If it is acceptable to simply fix the current code, we can provide patches. And if they could write some unittests that would be even better. :) Sure, it is a good thing to write tests. YO -- Yoshinori Okuji, Nexedi CTO Nexedi: Consulting and Development of Free / Open Source Software http://www.nexedi.com ERP5: Full Featured High End Open Source ERP http://www.erp5.com ERP5 Wiki: Developer Zone for ERP5 Community http://www.erp5.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
It has not worked for a long time, and that's because not many people use it. But those who do usually uses it a lot, so it would be bad if it goes away. I would propose that somone that uses this heavily (like the Nexedi people), joins the Zope Foundation as a commiter, gets commiter rights and holds his or her hands over this functionality. :) And if they could write some unittests that would be even better. :) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/ ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
On 10/27/06, Lennart Regebro [EMAIL PROTECTED] wrote: It has not worked for a long time, and that's because not many peopleuse it. But those who do usually uses it a lot, so it would be bad ifit goes away.I would propose that somone that uses this heavily (like the Nexedi people), joins the Zope Foundation as a commiter, gets commiter rightsand holds his or her hands over this functionality. :)He, thats odd, just yesterday at a local zope meeting, a friend mentioned that he knows a company that does the same thing quite regulary. Maybe they can be forces joined in getting it right in the zope core again!And if they could write some unittests that would be even better. :) --Lennart Regebro, Nuxeo http://www.nuxeo.com/CPS Content Management http://www.nuxeo.org/___ Zope-Dev maillist-Zope-Dev@zope.orghttp://mail.zope.org/mailman/listinfo/zope-dev**No cross posts or HTML encoding!** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Fwd: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)]
Hi!
I would like to resurrect this thread with my own recent experience.
My problem was to spread a set of zodb objects over several zope
installations. Every installation required some slight modifications to
be applied to original objects (such as usernames and passwords).
So I decided to export the original data in an xml dump and write a
small application which modifies the dump as necessary before import.
But to my distress I couldn't import neither a modified dump nor even
the original dump. The process failed with UnicodeDecodeError exception.
After some investigation I realized what was the problem (at least in my
case). The xml parser extracts all texts as unicode strings. But among
them are base64-encoded strings which are decoded into non-unicode
strings containing binary data. Of course this data can't be decoded by
any codec. The code in ppml.py sometimes concatenates the raw and
unicode strings and this raises UnicodeDecodeError.
I worked this around by converting the unicode strings into non-unicode
ones. Please look at the patch attached. Zope version 2.9.4.
--
Best regards,
Alexei
On 25. March 2006 21:40:48 +0100 Yoshinori Okuji yo at nexedi.com wrote:
On Saturday 25 March 2006 15:56, Andreas Jung wrote:
Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws
BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError.
I don't expect that the import functionality works for even more
complex
objects. So I consider the whole functionality as totally broken. The
generated XML might be useful to perform any processing outside
Zope but
using it for re-importing it into another Zope systems definitely does
_not_ work. So if the functionality should remain in Zope then it
should
be fixed
for Zope 2.10 lately.
Here is a quick patch for this problem (against 2.9.1). There were two
different problems:
- the id attributes were not generated, because the conditional was
reverse.
- unlike xmllib, expat always returns Unicode data, so simply
concatenating binary values generates Unicode objects with non-ascii
characters.
--- /tmp/ppml.py 2006-10-27 00:36:18.0 +0600
+++ /usr/lib/zope2.9/lib/python/Shared/DC/xml/ppml.py 2006-10-27 00:00:17.0 +0600
@@ -573,7 +573,9 @@
def save_tuple(self, tag, data):
T=data[2:]
if not T: return ')'
-return save_put(self, '('+string.join(T,'')+'t', data[1])
+try: ret='('+string.join(T,'')+'t'
+except UnicodeDecodeError: ret='('+string.join(map(str,T),'')+'t'
+return save_put(self, ret, data[1])
def save_list(self, tag, data):
L=data[2:]
@@ -590,7 +592,9 @@
D=data[2:]
if self.binary:
v=save_put(self, '}', data[1])
-if D: v=v+'('+string.join(D,'')+'u'
+if D:
+ try: v=v+'('+string.join(D,'')+'u'
+ except UnicodeDecodeError: v=v+'('+string.join(map(str,D),'')+'u'
else:
v=save_put(self, '(d', data[1])
if D: v=v+string.join(D,'s')+'s'
@@ -623,7 +627,8 @@
stop=string.rfind(x,'t') # This seems
if stop=0: x=x[:stop]# wrong!
v=save_put(self, v+x+'o', data[1])
-v=v+data[4]+'b' # state
+try: v=v+data[4]+'b' # state
+except UnicodeDecodeError: v=str(v)+str(data[4])+'b' # state
return v
def save_global(self, tag, data):
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
--On 25. März 2006 21:40:48 +0100 Yoshinori Okuji [EMAIL PROTECTED] wrote: On Saturday 25 March 2006 15:56, Andreas Jung wrote: Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError. I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately. Here is a quick patch for this problem (against 2.9.1). There were two different problems: - the id attributes were not generated, because the conditional was reverse. - unlike xmllib, expat always returns Unicode data, so simply concatenating binary values generates Unicode objects with non-ascii characters. Thanks for the patch (commited for Zope 2.8, 2.9. trunk). This solves at least the import problems on my side. Unfortunately there are no tests for export/import so fixing issues is like flying blindfolded. -aj pgpux7LfpurMA.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
--On 26. März 2006 09:58:07 +0100 Andreas Jung [EMAIL PROTECTED] wrote: Thanks for the patch (commited for Zope 2.8, 2.9. trunk). This solves at least the import problems on my side. Unfortunately there are no tests for export/import so fixing issues is like flying blindfolded. Although the patch does not help to export a complete Plone site as XML. This fails with: Traceback (innermost last): * Module ZPublisher.Publish, line 113, in publish * Module ZPublisher.mapply, line 88, in mapply * Module ZPublisher.Publish, line 40, in call_object * Module OFS.ObjectManager, line 547, in manage_exportObject * Module OFS.XMLExportImport, line 58, in exportXML * Module OFS.XMLExportImport, line 33, in XMLrecord * Module Shared.DC.xml.ppml, line 263, in load * Module pickle, line 872, in load * Module Shared.DC.xml.ppml, line 418, in load_binput AttributeError: 'unicode' object has no attribute 'id' -aj pgp5HnhJqBQvt.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
--On 24. März 2006 11:07:06 -0500 Jim Fulton [EMAIL PROTECTED] wrote: - it was broken multiple times in the past, it still has problems How so? The import functionality seem to be broken since at least Zope 2.7. I created a DTML method exported it and _tried_ to reimport it. Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError. I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately. Andreas pgpDsKQKSvaGQ.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
On Saturday 25 March 2006 15:56, Andreas Jung wrote:
Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws
BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError.
I don't expect that the import functionality works for even more complex
objects. So I consider the whole functionality as totally broken. The
generated XML might be useful to perform any processing outside Zope but
using it for re-importing it into another Zope systems definitely does
_not_ work. So if the functionality should remain in Zope then it should
be fixed
for Zope 2.10 lately.
Here is a quick patch for this problem (against 2.9.1). There were two
different problems:
- the id attributes were not generated, because the conditional was reverse.
- unlike xmllib, expat always returns Unicode data, so simply concatenating
binary values generates Unicode objects with non-ascii characters.
For the latter problem, I'm not sure if my patch is enough. But this patch
works with a simple dtml export/import.
YO
--
Yoshinori Okuji, Nexedi CTO
Nexedi: Consulting and Development of Free / Open Source Software
http://www.nexedi.com
ERP5: Full Featured High End Open Source ERP
http://www.erp5.com
ERP5 Wiki: Developer Zone for ERP5 Community
http://wiki.erp5.org
diff -urN Zope-2.9.1.orig/Dependencies/Shared-Zope-2.9.1/Shared/DC/xml/ppml.py Zope-2.9.1/Dependencies/Shared-Zope-2.9.1/Shared/DC/xml/ppml.py
--- Zope-2.9.1.orig/Dependencies/Shared-Zope-2.9.1/Shared/DC/xml/ppml.py 2006-03-15 17:11:00.0 +0100
+++ Zope-2.9.1/Dependencies/Shared-Zope-2.9.1/Shared/DC/xml/ppml.py 2006-03-25 21:31:25.183545415 +0100
@@ -414,14 +414,14 @@
def load_binput(self):
i = mloads('i' + self.read(1) + '\000\000\000')
last = self.stack[-1]
-if getattr(last, 'id', last) is not last:
+if getattr(last, 'id', last) is last:
last.id = self.idprefix + `i`
dispatch[BINPUT] = load_binput
def load_long_binput(self):
i = mloads('i' + self.read(4))
last = self.stack[-1]
-if getattr(last, 'id', last) is not last:
+if getattr(last, 'id', last) is last:
last.id = self.idprefix + `i`
dispatch[LONG_BINPUT] = load_long_binput
@@ -643,10 +643,10 @@
'pickle': lambda self, tag, attrs: [tag, attrs],
}
end_handlers={
-'pickle': lambda self, tag, data: data[2]+'.',
+'pickle': lambda self, tag, data: str(data[2])+'.',
'none': lambda self, tag, data: 'N',
'int': save_int,
-'long': lambda self, tag, data: 'L'+data[2]+'L\012',
+'long': lambda self, tag, data: 'L'+str(data[2])+'L\012',
'float': save_float,
'string': save_string,
'reference': save_reference,
___
Zope-Dev maillist - Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Deprecating XML export/import?
Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems: - the generated XML is pretty much too low-level to use it e.g. for migration issues - it was broken multiple times in the past, it still has problems - it has no maintainer, nobody wants to touch it without gloves I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import. Comments? Andreas pgpWVI8dK9rj4.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Deprecating XML export/import?
Hello, Em Sexta, 24 de Março de 2006 15:44, escreveu: I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import. I've exported to XML and edited the generated file by hand as a quick way to successfully import a site to another Zope instance, when I was getting import errors when using the .zexp. I'm not sure I'd be able to do that with the .zexp file. But then again, it was about the only time I really found a use for the XML export/import. Still, it was nice to have to have the possibility at the time. Nuno ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
Andreas Jung wrote: Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems: - the generated XML is pretty much too low-level to use it e.g. for migration issues I have been working on a Zope 2 project lately and have found XML export to be extremely usable. Templates and other large strings are enclosed in CDATA sections, which make them surprisingly readable. We've had sucess writing XSLT templates to transform the pickle data into formats easily parsable for particular applications. - it was broken multiple times in the past, it still has problems How so? - it has no maintainer, nobody wants to touch it without gloves Is that any more true than for lots of other things? I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import. Comments? -1 As I said above, I've been finding it surprisingly useful lately. When I first wrote it, I viewed it as someowhat of an academic exercise, but I've come to realize that it is far more useful than I originally thought. This is more so now that XSLT is widely used and quite capable of transforming the pickles into useful forms. Note that, IMO, there isn't likely to be a more generally useful form because usefullness is generally determined by a particular application. XSLT + XML pickles together provide a general mechanism that can adapt to particular application needs. I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote: I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. We have used it to produce a large backup before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEJBmxRAx5nvEhZLIRAonMAJ4m7YeEH8jqBKJk+orpkNzx3u+N/QCfRww8 4dZO6mwmUj0DqsQ5MketkjU= =U5Nk -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote: I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. We have used it to produce a large backup before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8. Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:12, Jim Fulton wrote: Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote: I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. We have used it to produce a large backup before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8. Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2. Yes, that's likely where the problem comes from. IMHO there is no reason to remove/deprecate it, but on the other hand it should not be advertised as a really user-friendly and easy way of exporting. It's still somewhat of a second class citizen when it comes to exporters. It might be in need of some warning stickers on the package ;) jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEJB3WRAx5nvEhZLIRAr3RAKCKomzIN1odFq/9fSfdWiet9m7r4ACguw/O EZRxyaVrtDb0NLCHO3RPp7U= =31YI -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:12, Jim Fulton wrote: Jens Vagelpohl wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote: I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. We have used it to produce a large backup before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8. Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2. Yes, that's likely where the problem comes from. IMHO there is no reason to remove/deprecate it, but on the other hand it should not be advertised as a really user-friendly and easy way of exporting. It's still somewhat of a second class citizen when it comes to exporters. It might be in need of some warning stickers on the package ;) Can you give an example of a first-class exporter? I'm not aware of a general solution. I guess there are solutions based on Archetypes. It should definately be advertized for what it is, but it is certainly much more than I realized in the past. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
On Fri, 24 Mar 2006, Jim Fulton wrote: Andreas Jung wrote: I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import. Comments? I filed a bug report (2051: xml import export fail) which was triggered by the need to move material from a recent zope (2.9) to an older zope (2.7.4) where the zexp format was incompatible. Eventually, I was able to make the XML work. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
--On 24. März 2006 11:07:06 -0500 Jim Fulton [EMAIL PROTECTED] wrote: Andreas Jung wrote: Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems: - the generated XML is pretty much too low-level to use it e.g. for migration issues I have been working on a Zope 2 project lately and have found XML export to be extremely usable. Templates and other large strings are enclosed in CDATA sections, which make them surprisingly readable. We've had sucess writing XSLT templates to transform the pickle data into formats easily parsable for particular applications. That's a good argument :-) - it was broken multiple times in the past, it still has problems How so? XML export did not work properly over some major Zope release (I think until Dieter fixed it). And there is at least one unicode related bug report pending (when I read such reports I am thinking about the following options: a) ignore the report and b) who the hell wrote the code and could fix it). - it has no maintainer, nobody wants to touch it without gloves Is that any more true than for lots of other things? Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-) As I said above, I've been finding it surprisingly useful lately. When I first wrote it, I viewed it as someowhat of an academic exercise, but I've come to realize that it is far more useful than I originally thought. I still have the impression that the implementation was made an academic excercie rather than something stable for production (.zexp works fine for production). This is more so now that XSLT is widely used and quite capable of transforming the pickles into useful forms. Note that, IMO, there isn't likely to be a more generally useful form because usefullness is generally determined by a particular application. XSLT + XML pickles together provide a general mechanism that can adapt to particular application needs. In this case it has to be fixed if it should be used a general mechanism. I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. Reasonable export/import should happen on a higher level. This low-level XML import/export is not really helpful when you want to migrate data between applications and frameworks. E.g. on the Plone/Archetypes level we have some mechanisms to export/import data defined through schemas (possibly we have something in Zope 3 (at least I once wrote a similar solution for Zope 3). The export/import mechanism is basically for moving data from one Zope instance to another instance. It should work in a reliable way and should not try to solve issues that should be solved on another level. To bring it to a point: further Zope 2 versions should get rid of buggy and esoteric features and apparently don't work for lots of ppl resulting in frustration and a bad image for Zope. Andreas --- - Andreas JungZOPYX Ltd. Co KG- - E-mail: [EMAIL PROTECTED] Web: www.zopyx.com, www.zopyx.de - --- pgpGcRTd8nHJz.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 24 Mar 2006, at 16:30, Jim Fulton wrote: Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2. Yes, that's likely where the problem comes from. IMHO there is no reason to remove/deprecate it, but on the other hand it should not be advertised as a really user-friendly and easy way of exporting. It's still somewhat of a second class citizen when it comes to exporters. It might be in need of some warning stickers on the package ;) Can you give an example of a first-class exporter? I'm not aware of a general solution. I guess there are solutions based on Archetypes. It should definately be advertized for what it is, but it is certainly much more than I realized in the past. That was just a metaphor. For me, only the standard binary zexp exporter qualifies as first class exporter at the moment because it just works. Apart from esoteric use cases like importing data into older ZODB implementations, which is highly questionable IMHO. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEJB+FRAx5nvEhZLIRAiUWAJ4807TgqX527sWgrZTTF7j0GoH2TQCfZDXj 0MrD1vpu79ERWnBCwu86NbY= =fpcK -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
Andreas Jung wrote: ... Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-) I don't see this as primarily a duplicate feature. I find the export aspect to be most compelling. Although, as others have pointed out, the openness of the XML format can be very useful for moving data between Zopes. Small data transformations (e.g. adjusting a class name) are possible and easy. As I said above, I've been finding it surprisingly useful lately. When I first wrote it, I viewed it as someowhat of an academic exercise, but I've come to realize that it is far more useful than I originally thought. I still have the impression that the implementation was made an academic excercie rather than something stable for production (.zexp works fine for production). It was an exercise. In some ways, it was ahead of it's time, as it is much more useful in the presense of XSLT. OTOH, there wasn't any intention to make it any lower quality than the binary format. This is more so now that XSLT is widely used and quite capable of transforming the pickles into useful forms. Note that, IMO, there isn't likely to be a more generally useful form because usefullness is generally determined by a particular application. XSLT + XML pickles together provide a general mechanism that can adapt to particular application needs. In this case it has to be fixed if it should be used a general mechanism. Sure I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. Reasonable export/import should happen on a higher level. This low-level XML import/export is not really helpful when you want to migrate data between applications and frameworks. I don't agree, if you are willing to use XSLT to extract the bits you want. E.g. on the Plone/Archetypes level we have some mechanisms to export/import data defined through schemas That's a good point. For Archetypes-based apps, this might be the way to go. I think that xml-pickle export is a reasonable approach for other apps. (possibly we have something in Zope 3 (at least I once wrote a similar solution for Zope 3). The export/import mechanism is basically for moving data from one Zope instance to another instance. It should work in a reliable way and should not try to solve issues that should be solved on another level. The unfinished fssync tool in Zope 3 is intended for this use case. It to uses an improved xmlpickle as a fallback. It would be nice to finish this, however, I can easily XSLT gun slingers prefering something like xml export. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
On Friday 24 March 2006 17:33, Andreas Jung wrote: - it has no maintainer, nobody wants to touch it without gloves Is that any more true than for lots of other things? Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-) We have been using XML export/import with ERP5[1] in many production systems for a long time. Only this feature allows us to understand changes among different versions in a human readable format, and even to edit the contents by ordinary text editors. Without this feature, it is nearly impossible to work on large systems in a distributed way. XML export/import is so valuable to make a bridge between TTW development and distributed development with the technology Business Template[2,3]. To make the feature more convenient, we have locally developed some monkey patches[4,5] to Zope so that: - the output is more stable (i.e. the order of tags and the reference numbers are more stable) to make diff usable for human - unicode objects work fine, assuming that data should be encoded in UTF-8 We would be happy to submit our patches, if you think they are generally useful. [1] http://www.erp5.org [2] http://wiki.erp5.org/HowToCreateBusinessTemplates [3] http://wiki.erp5.org/HowToCreateBusinessTemplates [4] http://cvs.erp5.org/ERP5Type/patches/XMLExportImport.py [5] http://cvs.erp5.org/ERP5Type/patches/ppml.py YO -- Yoshinori Okuji, Nexedi CTO Nexedi: Consulting and Development of Free / Open Source Software http://www.nexedi.com ERP5: Full Featured High End Open Source ERP http://www.erp5.com ERP5 Wiki: Developer Zone for ERP5 Community http://wiki.erp5.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
--On 24. März 2006 18:09:36 +0100 Yoshinori Okuji [EMAIL PROTECTED] wrote: On Friday 24 March 2006 17:33, Andreas Jung wrote: - it has no maintainer, nobody wants to touch it without gloves Is that any more true than for lots of other things? Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-) We have been using XML export/import with ERP5[1] in many production systems for a long time. Only this feature allows us to understand changes among different versions in a human readable format, and even to edit the contents by ordinary text editors. Without this feature, it is nearly impossible to work on large systems in a distributed way. XML export/import is so valuable to make a bridge between TTW development and distributed development with the technology Business Template[2,3]. As I said, I am open to arguments and this convinces that the XML export/import is useful for some ppl and should stay. I am not dogmatic on such issues but such a discussion help to wake people but a bit To make the feature more convenient, we have locally developed some monkey patches[4,5] to Zope so that: - the output is more stable (i.e. the order of tags and the reference numbers are more stable) to make diff usable for human - unicode objects work fine, assuming that data should be encoded in UTF-8 We would be happy to submit our patches, if you think they are generally useful. Of course we do appreciate any kind of patches as long they make sense and have reasonable test to document the intended behavior. In general patches are always welcome :-) Andreas pgpJ4LyjD2r2f.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
Yoshinori Okuji wrote: On Friday 24 March 2006 17:33, Andreas Jung wrote: - it has no maintainer, nobody wants to touch it without gloves Is that any more true than for lots of other things? Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-) We have been using XML export/import with ERP5[1] in many production systems for a long time. Only this feature allows us to understand changes among different versions in a human readable format, and even to edit the contents by ordinary text editors. Without this feature, it is nearly impossible to work on large systems in a distributed way. XML export/import is so valuable to make a bridge between TTW development and distributed development with the technology Business Template[2,3]. To make the feature more convenient, we have locally developed some monkey patches[4,5] to Zope so that: - the output is more stable (i.e. the order of tags and the reference numbers are more stable) to make diff usable for human I did quite a bit of work along these lines for Zope 3. Perhaps we can find a way to share the Z2 and Z3 code. - unicode objects work fine, assuming that data should be encoded in UTF-8 I think the Z3 xml pickle code handles Unicode too. We would be happy to submit our patches, if you think they are generally useful. Perhaps first, we should try the Zope 3 version of xml pickle. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: XML export/import is cool! (Was Re: [Zope-dev] Deprecating XML export/import?)
On 3/24/06, Jim Fulton [EMAIL PROTECTED] wrote: We've had sucess writing XSLT templates to transform the pickle data into formats easily parsable for particular applications. As part of a recent task (likely the same one Jim's referring to here!), I transformed the XML export into another XML pickle, but for a simpler data structure that could be unpickled without dealing with the application classes. The CDATA marked sections that held the templates aren't useful for XML tools, but they do make it somewhat easier to read the pickles directly (valuable when trying to figure out what a transform is starting from!). -Fred -- Fred L. Drake, Jr.fdrake at gmail.com Don't let schooling interfere with your education. -- Mark Twain ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Deprecating XML export/import?
Andreas Jung wrote at 2006-3-24 16:44 +0100: Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems: - the generated XML is pretty much too low-level to use it e.g. for migration issues - it was broken multiple times in the past, it still has problems There was a single error. True, for a longer period -- until Tres provided the tiny fix ... I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. What will remain of Zope in the longer run? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Deprecating XML export/import?
--On 24. März 2006 21:05:12 +0100 Dieter Maurer [EMAIL PROTECTED] wrote: ... I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. What will remain of Zope in the longer run? The good things that work :-) In the past with every major release some problems with old and scary components came up. In this situation you have basically three options: - ignore such problems - fix the code - rewrite the code I have a bad feeling (as release manager) about ignoring problems and releasing stuff. Fixing and rewriting code often requires a deep understanding of the concepts, the implementation especially if there is no documentation and no tests. In many cases Jim is the original author or knows at least of the concepts since many of the original Zope developers are no longer with ZC or even part of the Zope community. So as a consequence in the long term we should get rid of stuff that is either broken or considered obsolete by the majority of the developers and for the majority of the developers. Figuring out what the majority is in a particular case is of course always a problem Returning with the discussion to ZClasses (also off-topic in this XML export/import thread): they blocked the 2.8.0 release for a long time. If there is an unimportant second-class or third-class component of Zope blocking a major release in the future than it should be subject to be deprecated (or even removed if necessary). Andreas --- - Andreas JungZOPYX Ltd. Co KG- - E-mail: [EMAIL PROTECTED] Web: www.zopyx.com, www.zopyx.de - --- pgpdmEiPlGNlP.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
