Adam Groszer wrote:
What about pushing the problem then to the lower level, to Python
itself. I think all developers are fighting the same problem, so all
Python developers would benefit from the solution. As I know (that may
be wrong) not many even if any language supports that, so that would
make one big plus point on the Python side also.

As I don't have really deep knowledge of the Python interpreter
itself, I cannot imagine how weird is the idea. Maybe we should ask
Guido to have some thoughts about that.

I've spent time thinking about this. Modern operating systems are surprisingly good at reloading processes, but in general, it's hard to reload pieces of a process. What's the difference?

I think the difference is in the type of interdependence. Operating systems force processes to talk to each other through high level mechanisms like files, streams, sockets, memory mapped I/O, and so on. Good programmers understand that processes can die and thus make their software resilient to communication channel interruptions.

Within a process, programmers have no such expectation. Once the programmer imports a module, the programmer expects the imported module to remain unchanged. There is rarely any concept that modules are actually communicating with each other. A sticky morass of inter-module pointers quickly forms, leaving little hope of reliably reloading arbitrary modules. The operating system has to intervene in order to start the process over.

Shared memory makes it possible to link processes at a deeper level, but in practice, shared memory is used mostly for threading. It's no coincidence that multiple threads are generally thought of as a single process that has to restart together. Once two processes share pointers, it's hard to unbind them.

So I have considered two basic approaches for reliably reloading a module:

1) Code the reloadable module as a pure communication endpoint, treating the module almost like a process. No other modules should import from the module; instead, the module should register itself with a framework and other modules should talk to the module only through that framework. This is a good approach for writing reloadable application-specific plugins. You can also support clusters of modules that represent a single plugin.

The Zope 2 refresh mechanism works quite well with products written this way. Unfortunately, keeping modules free of interdependencies is difficult, and that's a major support risk.

2) Make reloadable code fundamentally different. If module X is supposed to be reloadable, and X creates a module-level global variable Y, and module Z imports Y, then Y needs to be decorated in such a way that Z's view of Y can change automatically when X is reloaded.

This second approach has subtle limitations, though. What if Y has the value 10 and Z defines a global variable A whose value is (Y**2)? The value of A might need to change when Y changes, but how can we arrange for that to happen without making a mess of the code? I doubt there's any reasonable general solution.

Even more subtle is what happens when a reloadable module holds a registry of things imported from other modules. When the module is reloaded, should the registry get cleared? Zope 2's refresh says the registry should be cleared, but in practice, this confuses everyone.

To solve this, I think reloadable modules need to have a special global namespace. Everything in the global namespace, as well as everything reachable from the global namespace, must be explicit about what happens at the time another module imports it or the module is reloaded. I think this could make a refresh mechanism like the one in Zope 2 reliable. It has a lot of similarity with persistent modules, but it might be simpler. I haven't thought it all the way through. The idea came to me about halfway through this post. :-)

Zope3-dev mailing list

Reply via email to