Re: [Zope] Re: Zope 2 and Zope 3

[Paul Winkler]

On Thu, Feb 15, 2007 at 11:50:38AM +0100, Philipp von Weitershausen wrote:
> What's already possible is to have a minimal ZODB with only one 
> persistent object: a SQLObject or SQLAlchemy container. That's a 
> container (e.g. like a folder) whose items aren't persisted in the ZODB 
> but come from a relational database.

This works fine in a Zope 2 / Five world too.  We have apps that have
nothing in the ZODB but a Z*DA instance.  Then we have zope 3 views
and adapters that acquire that instance and use it to talk to the
database, using ZSQL to build the queries. We haven't drunk the ORM
cool-aid yet. :)

In other words: The ZODB may be there but you can pretty much ignore
it if you don't need it :)

> You seem to already have come to the conclusion that having code live in 
> both the filesystem and the ZODB can be painful. I think a good first 
> step for you would be to migrate your remaining ZODB-based code to the 
> filesystem. That not only makes deployment easier, you're also free to 
> refactor it then (e.g. using Zope 3 idioms). In the long run, this also 
> means saying good-bye to things like External Methods because they 
> require code on the filesystem and configuration in the ZODB.

+1

Here's my off-the-cuff attempt at "How To Safely Move A Big Legacy App
Out of the ZODB".

First, get and install FSDump (google for it).

Add a Dumper to a folder containing DTML (it works recursively on
child folders too), configure the Dumper, click the "Save and dump"
button, and voila, there's your code on the filesystem.  It'll store
properties too, in files named like *.metadata.

Then you need a way to use the stuff from the filesystem.
FileSystemSite would be ideal, because it understands those .metadata
files: http://www.infrae.com/download/FileSystemSite

Using it is a bit non-obvious if you're not familiar with CMF, from
which it was extracted.  You'll need to create a minimal Product on
the filesystem that consists of:

- a directory in Products, let's call it Products/mystuff
- an __init__.py at Products/mystuff/__init__.py that looks like this:

   from Products.FileSystemSite.DirectoryView import registerDirectory
   registerDirectory('mysubdir, globals())

- a subdirectory at Products/mystuff/mysubdir.  Put the files and
  directories you dumped in here.

That's it. Restart Zope, add an instance of Filesystem Directory View,
and you'll be prompted for the directory to choose; the only choice
will be the "mysubdir" you registered above. Click OK and you're done.

Check this new Product into source control and rejoice.

One gotcha - files ending in .dtml will be treated as DTML Methods,
not DTML Documents. So if you relied on the semantic difference
between those, you will have some issues to sort out.

Another gotcha - you'll need another plan to deal with those External
Methods, since AFAIK neither Dumper nor FileSystemSite will handle
them.

One expedient if slightly tedious technique would be to just move the
External Method files out of Extensions/ and into
$INSTANCE_HOME/lib/python/.  Then use calls to allow_module() so they
can be imported by through-the-web code (google will tell you more
about that).  Then, delete each external method from the ZMI and in
its place add a Script (Python) with the same id.  Such a script would
have a body something like:

   from my_ext_method_module import myExternalMethodFunction
   return myExternalMethodFunction(arguments)

Once that's done, those scripts can be Dumped into your filesystem
code, and all the DTML that includes calls to them should work.

A final gotcha - if you have ZClasses, I can't help you :)


At this point, you have the same application you started with, it's
just on the filesystem.  Not as clean as a rewrite, but a hell of a
lot more expedient - faster and less risky.

Next you'll want to start adding regression tests to your product
(ZopeTestCase and Zelenium might be useful tools). Then you have a
safety net so you can confidently start refactoring as needed.  *This*
is when you can start thinking about rewriting stuff :-)

I'd put this email somewhere on zopewiki.org, but it seems to be down
at the moment (Bad Gateway).

> It is certainly possible to execute filesystem-based DTML (using
> Globals.DTMLFile).

Right.  But that's a bit of a pain when you have dozens or hundreds of
DTML methods and your first priority is just to get stuff out of the
ZODB and into source control without breaking anything.

-- 

Paul Winkler
http://www.slinkp.com
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] Re: Zope 2 and Zope 3

[Philipp von Weitershausen]

Geoff Gerrietts wrote:
[some paragraphs about scaling which I can't really comment on]


As for source control, I figure all of your code (DTML, yuck) is in the
ZODB. This went out of fashion a long time ago, most serious development
happens on the filesystem (in Python packages) which can obviously be
source-controlled very well.


A good deal of the code is in the ZODB, because it's (yuck) DTML
documents, yes. We have another sizable chunk that lives on the
filesystem. The challenge in managing the build/deploy process has
been trying to find a way to keep those two conjoined. I suspect that
any dependency on the ZODB at all is likely to be considered an
impediment, and my quick eyeballing says it's not gone or even really
optional under Zope 3 (but I'm sure we could work something out.)


The ZODB is still there in Zope 3, but Zope 3 is much less dependent on 
it. In fact, there are people who are using Zope 3 without any ZODB 
instance at all, but directly talk to a relational database. Currently 
this requires a bit of hacking, but our good man Chris Withers has 
expressed interest in making the next release of Zope 3 more flexible in 
this regard.


What's already possible is to have a minimal ZODB with only one 
persistent object: a SQLObject or SQLAlchemy container. That's a 
container (e.g. like a folder) whose items aren't persisted in the ZODB 
but come from a relational database.


You seem to already have come to the conclusion that having code live in 
both the filesystem and the ZODB can be painful. I think a good first 
step for you would be to migrate your remaining ZODB-based code to the 
filesystem. That not only makes deployment easier, you're also free to 
refactor it then (e.g. using Zope 3 idioms). In the long run, this also 
means saying good-bye to things like External Methods because they 
require code on the filesystem and configuration in the ZODB.



Regarding "oh you'll hafta start over", it's pretty much true, if you
want to switch to Zope 3. But nobody says you have to. You can do it
incrementally by porting some of your app's functionality to Zope 3
components step by step (as mentioned already, most work in Zope 2). Big
projects like Plone aren't rewriting their whole codebase either, but
new development is completely Zope3-based. Their target platform still
is Zope 2, though.


"You'll hafta start over" is only ever true in degrees, though, and
I'm still trying to figure out what the degrees are. The business
logic will still be the same. Can we hack together a DTML processor
that allows us to export the DTML documents to the filesystem and
publish them from there? Maybe, I don't know. Did someone else already
do that? Don't know that either. How dramatically will our products
need to change? Probably 75% of our Python code is written in a
bastardized form of ExternalMethod; we might be able to leverage that
unfortunate architectural choice to significant advantage during our
porting phase. These are the kinds of questions I have, and I think
the answers probably aren't easy or someone would have offered them
up.


It is certainly possible to execute filesystem-based DTML (using 
Globals.DTMLFile). It's hard to say how much your products will have to 
change without knowing the codebase. If you have external methods, it 
shouldn't be too hard to refactor this code into components in "proper" 
Python modules. To make things easy, the external methods could stay at 
first, for backwards compatibility. I would take this one step at a time 
(that's at least what everybody else seems to do and it looks like it's 
working out for everybody).



Someone's going to need to learn enough Zope 3 to answer the
questions. I'm not sure it will be me, but maybe. :)


Well, whoever it'll be, there are books (e.g. http://worldcookery.com) 
and trainings (e.g. http://trizpug.org/boot-camp/camp5) that should help 
with the learning :).



--
http://worldcookery.com -- Professional Zope documentation and training
Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

[Zope] Re: Zope 2 and Zope 3

[Geoff Gerrietts]

On 2/14/07, Philipp von Weitershausen <[EMAIL PROTECTED]> wrote:


There will definitely be a Zope 2.11 and at this point I see no reason
why there shouldn't be a Zope 2.12. We (the Zope Community, not
necessarily Zope Corporation) will maintain Zope 2 as long as it's
necessary. "Maintaining" in this case also means integrating more Zope 3
technologies with each release. Nowadays (Zope 2.10) you can already
write applications, or at least base libraries, that work on both Zope 3
and Zope 2.


This is very helpful for setting a planning horizon, thank you.


I'm surprised you're experiencing poor scaling. I think Zope can scale
pretty well, or at least it can be *made* to scale pretty well. It all
depends on your setup and, of course, on your application.


I can't swear to the accuracy of what I'm about to say but it is an
operating hypothesis that describes accurately behavior that we are
seeing.

Our application is highly dynamic, by which I mean that satisfying any
given request may require a great deal of database activity. The
database in question is almost always Oracle; I'm not counting the
ZODB accesses, here.

Oracle will handle a great many concurrent requests fairly gracefully,
but its average transaction time is not always excellent, especially
for moderately complicated queries. While most queries are sub-second,
populating a page that requires multiple queries to fill in all its
data fields, or even populating a page with only one or two
complicated queries, can require greater than a single second of
database I/O. This cost is offset by the ability to have other threads
processing while your first thread waits for the database to come
back.

Zope likes to have 4 threads processing requests. This preference is
documented in several places, and experiments have historically borne
out the "4 thread hypothesis". We have been unable to realize
significant performance improvements by increasing the number of
available worker threads, but we are able to realize performance
improvements by pointing two Zope servers against the same database.

Zope isn't using much CPU, and it's not using much memory, but it
peaks out at 4 concurrent requests; everything else goes in a queue
behind them. If those 4 concurrent requests happen to take 4 or 5
seconds each, the queue builds deep behind them, possibly with very
simple one-offs.

Historically, we've addressed this by sticking another Zope out there.
We have plans to run multiple Zope instances on a single machine, like
they do (did?) at plone.org; this still requires configuration
management work and testing. But the feeling is that we have a lot
more hardware than we really should need, to handle the volumes we're
dealing with, and we have that hardware because we need more
"channels" to handle our concurrent traffic load.


As for source control, I figure all of your code (DTML, yuck) is in the
ZODB. This went out of fashion a long time ago, most serious development
happens on the filesystem (in Python packages) which can obviously be
source-controlled very well.


A good deal of the code is in the ZODB, because it's (yuck) DTML
documents, yes. We have another sizable chunk that lives on the
filesystem. The challenge in managing the build/deploy process has
been trying to find a way to keep those two conjoined. I suspect that
any dependency on the ZODB at all is likely to be considered an
impediment, and my quick eyeballing says it's not gone or even really
optional under Zope 3 (but I'm sure we could work something out.)



Regarding "oh you'll hafta start over", it's pretty much true, if you
want to switch to Zope 3. But nobody says you have to. You can do it
incrementally by porting some of your app's functionality to Zope 3
components step by step (as mentioned already, most work in Zope 2). Big
projects like Plone aren't rewriting their whole codebase either, but
new development is completely Zope3-based. Their target platform still
is Zope 2, though.


"You'll hafta start over" is only ever true in degrees, though, and
I'm still trying to figure out what the degrees are. The business
logic will still be the same. Can we hack together a DTML processor
that allows us to export the DTML documents to the filesystem and
publish them from there? Maybe, I don't know. Did someone else already
do that? Don't know that either. How dramatically will our products
need to change? Probably 75% of our Python code is written in a
bastardized form of ExternalMethod; we might be able to leverage that
unfortunate architectural choice to significant advantage during our
porting phase. These are the kinds of questions I have, and I think
the answers probably aren't easy or someone would have offered them
up.

Someone's going to need to learn enough Zope 3 to answer the
questions. I'm not sure it will be me, but maybe. :)

Thanks for all the insight, I'm making plans and pleas and I have a
direction for my research now. :)

Thanks,
--G.

[Zope] Re: Zope 2 and Zope 3

[Philipp von Weitershausen]

Geoff Gerrietts wrote:
I have a couple questions about the current game plan for Zope 
development that I was unable to answer for myself by searching through 
mailing list archives and the zope.org  website. I hope 
my inability to answer them is not merely a symptom of poor search term 
choices, and I hope someone can answer them for me.


You're right, it is a bit tricky to google for this.

Zope 3 release notes usually contain the following notice (from 
http://mail.zope.org/pipermail/zope-announce/2006-September/002016.html):


  Zope 3 is the next major Zope release and has been written from
  scratch based on the latest software design patterns and the
  experiences of Zope 2.

  Cleanup of the Zope 3 packages has continued to ensure a flexible and
  scalable platform. We continued the work on making the transition from
  Zope 2 to Zope 3 by making Zope 2.10 use even more of the Zope 3
  packages. But the transition is far from complete yet. **You can't run
  Zope 2 applications in Zope 3.**

I've posted a couple times before (a long time ago) about the large web 
application I work on; it's many hundreds of KLOC and an unfortunately 
large percentage of it is DTML. It's still more or less in that state; 
the realities of being a for-profit business in a rapidly evolving 
marketplace permit very little time for maintenance or aggressive 
refactoring of the application. That's the background for my questions; 
I'm trying to figure out how loudly I can shout "Impending doom!" to 
upper management without endangering my credibility.


1. Last time I talked to anyone from Zope Corp, the plan was to quiesce 
Zope 2 in favor of Zope 3. At that time, I think I remember people 
saying 2.9 would be the last in the Zope 2 series. Now there's a 2.10, 
so clearly it wasn't. How long does a business like mine have before 
there is no clear upgrade path in the Zope 2 series? Is 2.10 now the 
final Zope 2 release, or will there be a 2.11 and 2.12?


There will definitely be a Zope 2.11 and at this point I see no reason 
why there shouldn't be a Zope 2.12. We (the Zope Community, not 
necessarily Zope Corporation) will maintain Zope 2 as long as it's 
necessary. "Maintaining" in this case also means integrating more Zope 3 
technologies with each release. Nowadays (Zope 2.10) you can already 
write applications, or at least base libraries, that work on both Zope 3 
and Zope 2.


2. Whenever someone asks about moving an existing application from Zope 
2 to Zope 3, there's a round of flip "oh you'll hafta start over" 
responses. I understand that the two architectures are fundamentally 
different, and that I'll need to rebuild my object database, etcetera. 
Am I also correct in believing that all products must be completely 
rewritten, and that DTML is no longer a supported templating language? 
Has the users/roles/permissions security structure disappeared or been 
supplanted? I am trying to ascertain at a glance how much we will need 
to rebuild for ourselves; if it's enough then the problems we have had 
with Zope (poor scaling, poor integration with source control) may 
outweigh the advantages of remaining on the platform.


I'm surprised you're experiencing poor scaling. I think Zope can scale 
pretty well, or at least it can be *made* to scale pretty well. It all 
depends on your setup and, of course, on your application.


As for source control, I figure all of your code (DTML, yuck) is in the 
ZODB. This went out of fashion a long time ago, most serious development 
happens on the filesystem (in Python packages) which can obviously be 
source-controlled very well.


Regarding "oh you'll hafta start over", it's pretty much true, if you 
want to switch to Zope 3. But nobody says you have to. You can do it 
incrementally by porting some of your app's functionality to Zope 3 
components step by step (as mentioned already, most work in Zope 2). Big 
projects like Plone aren't rewriting their whole codebase either, but 
new development is completely Zope3-based. Their target platform still 
is Zope 2, though.



--
http://worldcookery.com -- Professional Zope documentation and training
Next Zope 3 training at Camp5: http://trizpug.org/boot-camp/camp5

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )