Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?
On 2007-10-26 05:41, Barry Warsaw wrote: > On Oct 22, 2007, at 11:30 PM, [EMAIL PROTECTED] wrote: > >> It's not clear that any of these implementations is going to be >> perfect. >> Maybe none ever will be. > > I would agree with this. You write a program and know you need to > implement some kind of resource locking, so you start looking for > some OTS solution. But then you realize that your application needs > somewhat different semantics or needs to work in platforms or > environments that the OTS code doesn't handle. Just a few days ago, > I was looking at some locking code that needed to work across > multiple invocations of a script on multiple machines, and the only > thing they shared was a PostgreSQL connection, so we ended up wanting > to use its advisory locks. > >> In his reply Jean-Paul made this comment: > >> It might be nice to have something like that in the standard >> library, >> but it's very simple once you know what to do. > >> I'm not so sure about the "very simple" part, especially if you aren't >> familiar with all the ins and outs of the different platforms. > > I'd totally agree with this. Locking seems simple, but it's got some > really tricky aspects that need to be coded just right or you'll be > in a world of hurt. Mailman's LockFile.py (which you're right is > *nix only) is stable now, but has had some really subtle bugs in the > past. You might want to take a look at the FileLock.py module that's part of the eGenix mx Base distribution (mx.Misc.FileLock). It works reliably on Unix and Windows, doesn't rely on fcntl and has been in use for years. The only downside is that it's application specific, ie. only applications using the module for locking will detect the locks - but then again: this is exactly the problem you typically want to solve. >> The fact >> that the first three bits of code I was referred to were >> implemented by >> three significant Python tools/platforms and that all are different >> in some >> significant ways suggests that there is some both an underlying >> need for a >> file locking mechanism but with a lack of consensus about the best >> way to >> implement the mother-of-all-file-locking schemes for Python. Maybe >> the best >> place for this is in the distribution. PEP? > > I don't think any one solution will work for everybody. I'm not even > sure we can define a common API a la the DBAPI, but if something were > to make it into the standard distribution, that's the direction I'd > go in. Then we can provide various implementations that support the > LockingAPI under various environments, constraints, and platforms. > If we wanted to distribute them in the stdlib, we could put them all > in a package and let the user decide which features they need. > > I'm still planning on de-Mailman-ifying LockFile.py sometime soon. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 26 2007) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Deadlock by a second import in a thread
Facundo Batista wrote:
> Feel free to do it. But note, that some imports are inside the call()
> function, this could have more implications that you see (at least I
> saw) at first glance.
CC to get Guido's attention
First of all I don't understand what you mean with "that some imports
are inside the call() function". Please elaborate on it.
I skimmed through the code and found just a handful of modules that are
important by name as well as imported more than once through the life
time of a python process. The most noticeable modules are time,
_strptime, resource, unicodedata and warnings. The other modules like
__builtin__, zlib and warnings are just loaded once or twice during the
boot strapping of the interpreter.
Guido:
What do you think about storing the modules in an "extern PyObject
*PyMod_Spam" variable? I could either store them when they are used the
first time or I could load them in Py_InitializeEx.
Christian
$ find -name '*.c' | xargs grep PyImport_ImportModule\(\"
./Objects/unicodeobject.c:m =
PyImport_ImportModule("unicodedata");
./Objects/exceptions.c:bltinmod = PyImport_ImportModule("__builtin__");
./PC/bdist_wininst/install.c: mod = PyImport_ImportModule("__builtin__");
./Modules/_ctypes/callbacks.c: mod = PyImport_ImportModule("ctypes");
./Modules/_ctypes/callbacks.c: mod = PyImport_ImportModule("ctypes");
./Modules/cPickle.c:if (!( copy_reg =
PyImport_ImportModule("copy_reg")))
./Modules/cPickle.c:if (!( t=PyImport_ImportModule("__builtin__")))
return -1;
./Modules/posixmodule.c:PyObject *m =
PyImport_ImportModule("resource");
./Modules/zipimport.c: zlib = PyImport_ImportModule("zlib");
/* import zlib */
./Modules/datetimemodule.c: PyObject *time =
PyImport_ImportModule("time");
./Modules/datetimemodule.c: PyObject *time =
PyImport_ImportModule("time");
./Modules/datetimemodule.c: time = PyImport_ImportModule("time");
./Modules/datetimemodule.c: if ((module =
PyImport_ImportModule("time")) == NULL)
./Modules/timemodule.c:PyObject *strptime_module =
PyImport_ImportModule("_strptime");
./Modules/timemodule.c: m = PyImport_ImportModule("time");
./Modules/gcmodule.c: tmod = PyImport_ImportModule("time");
./Modules/main.c: runpy = PyImport_ImportModule("runpy");
./Modules/main.c: v = PyImport_ImportModule("readline");
./Modules/parsermodule.c:copyreg = PyImport_ImportModule("copy_reg");
./Modules/_cursesmodule.c: PyObject *m = PyImport_ImportModule("curses");
./Python/mactoolboxglue.c: m = PyImport_ImportModule("MacOS");
./Python/pythonrun.c: warnings_module = PyImport_ImportModule("warnings");
./Python/pythonrun.c: PyObject *bimod =
PyImport_ImportModule("__builtin__");
./Python/pythonrun.c: m = PyImport_ImportModule("site");
./Python/errors.c: mod = PyImport_ImportModule("warnings");
./Python/import.c: zimpimport = PyImport_ImportModule("zipimport");
./Doc/tools/sphinx/jinja/_speedups.c: PyObject *datastructure =
PyImport_ImportModule("jinja.datastructure");
./Mac/Modules/MacOS.c: m = PyImport_ImportModule("macresource");
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Deadlock by a second import in a thread
2007/10/26, Christian Heimes <[EMAIL PROTECTED]>:
> First of all I don't understand what you mean with "that some imports
> are inside the call() function". Please elaborate on it.
I'm talking about the "call" function defined in the _sre.c file. This
function has a call to PyImport_Import() inside it.
In the "bug" I was persuing, this call to PyImport_Import() causes the
deadlock after "call" being called from line 2314:
filter = call(
SRE_PY_MODULE, "_subx",
PyTuple_Pack(2, self, ptemplate)
);
I just was saying the calls to "imports" are so deep, that you need to
touch a lot of code to make sure that they're executed in a sane way.
With "sane way", I mean that the imports should be executed when the
user makes "import foobar". Importing thing when the user makes
time.strptime() or re.sub() is breaking the "least surprise" rule,
which is a bad thing especially when talking of imports, that can
cause (and actually do!) deadlocks.
> I skimmed through the code and found just a handful of modules that are
> important by name as well as imported more than once through the life
> time of a python process. The most noticeable modules are time,
> _strptime, resource, unicodedata and warnings. The other modules like
> __builtin__, zlib and warnings are just loaded once or twice during the
> boot strapping of the interpreter.
This goes along what I said in my first mail: everytime you do
time.strptime(), that function tries to import "_strptime.py", which
is, at least, inefficient.
> What do you think about storing the modules in an "extern PyObject
> *PyMod_Spam" variable? I could either store them when they are used the
> first time or I could load them in Py_InitializeEx.
You shouldn't import them when they're used in the first time, are you
still are breaking the least surprise rule (see last parragraph).
However, one possible way to solve this problems is to, in every
module, import everything the module will ever need at init time. Note
that you actually do not need ane extern variable here, as you won't
be accessing it from other files, making it global and static would be
ok.
I was about to commit the following change in timemodule.c, that is a
good example of what I just said (I didn't commit it, because I'm not
full aware of the issue that Brett described):
1. I extracted the import from the strptime function:
@@ -514,13 +517,11 @@
static PyObject *
time_strptime(PyObject *self, PyObject *args)
{
-PyObject *strptime_module = PyImport_ImportModule("_strptime");
PyObject *strptime_result;
if (!strptime_module)
return NULL;
strptime_result = PyObject_CallMethod(strptime_module,
"strptime", "O", args);
-Py_DECREF(strptime_module);
return strptime_result;
}
2. Created a global static variable that will hold the module:
@@ -98,6 +98,9 @@
/* For Y2K check */
static PyObject *moddict;
+/* This will be initializied at module init time */
+static PyObject *strptime_module;
+
/* Exposed in timefuncs.h. */
time_t
_PyTime_DoubleToTimet(double x)
3. Imported the module at init time:
@@ -848,6 +849,8 @@
Py_INCREF(&StructTimeType);
PyModule_AddObject(m, "struct_time", (PyObject*) &StructTimeType);
initialized = 1;
+
+strptime_module = PyImport_ImportModule("_strptime");
}
Regards,
--
.Facundo
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SSL 1.10, to fix client-side bug
I've updated the PyPI SSL package to 1.10, to fix a fairly serious bug in sendall() I found while updating the trunk code. http://pypi.python.org/pypi/ssl/ Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Tracker Issues
ACTIVITY SUMMARY (10/19/07 - 10/26/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1311 open (+18) / 11509 closed (+18) / 12820 total (+36) Open issues with patches: 414 Average duration of open issues: 682 days. Median duration of open issues: 759 days. Open Issues Breakdown open 1307 (+18) pending 4 ( +0) Issues Created Or Reopened (37) ___ Py3K cannot run as ``python -S`` 10/21/07 CLOSED http://bugs.python.org/issue1267reopened gvanrossum py3k, patch Fixes for profile/cprofile 10/19/07 CLOSED http://bugs.python.org/issue1302created tiran py3k, patch adapt str8 constructor to bytes constructor 10/19/07 CLOSED http://bugs.python.org/issue1303created georg.brandl py3k, patch py3k compilation on Windows 10/19/07 CLOSED http://bugs.python.org/issue1304created amaury.forgeotdarc patch socket.py hangs on Mac OS X 10/20/07 CLOSED http://bugs.python.org/issue1305created andres Embedded python reinitialization 10/20/07 http://bugs.python.org/issue1306created Karemir smtpd.SMTPServer throws exception on MAIL command with no arg10/23/07 CLOSED http://bugs.python.org/issue1307reopened gvanrossum patch unicode(None) anomaly10/21/07 CLOSED http://bugs.python.org/issue1308created jgsack windows build fix10/22/07 CLOSED http://bugs.python.org/issue1309created tiran patch tempfile breaks on Windows 10/22/07 CLOSED http://bugs.python.org/issue1310created tiran patch os.path.exists(os.devnull) regression on windows 10/22/07 http://bugs.python.org/issue1311created d_kagedal doctest EXCEPTION_RE can't handle preceding output 10/22/07 http://bugs.python.org/issue1312created nouri patch Typo in Python Library Reference documentation: logging module 10/22/07 CLOSED http://bugs.python.org/issue1313created nhaines Patch for strftime problem on German Windows 10/23/07 CLOSED http://bugs.python.org/issue1314created tiran patch Bug in pstats.print_callers 10/23/07 http://bugs.python.org/issue1315created JoelSchaerer fix a few PyInt_Check vs PyLong_Check problems 10/23/07 CLOSED http://bugs.python.org/issue1316created georg.brandl py3k, patch smtplib.SMTP docs10/23/07 http://bugs.python.org/issue1317created fdrake Remove os.tmpnam() and os.tempnam() 10/23/07 CLOSED http://bugs.python.org/issue1318created tiran patch py3k: fixes for test_ctypes 10/23/07 CLOSED http://bugs.python.org/issue1319created amaury.forgeotdarc patch PCBuild8 Solution Support Changes10/24/07
Re: [Python-Dev] Python developers are in demand
I come from a corporate IT background working as a drone in large businesses, starting about 30 years ago (argh!). I come from an IBM mainframe background, but with constant exposure and interfaces with other platforms, and writing code with a variety of languages and scripting languages throughout my career. I recently completed a very long and painful transition to becoming at least a part time Java developer. For the past several years I have had a background conversation with myself on the topic of the Perfect Programming Language; what would it look like/feel like. Then I stumbled on Python, and I am totally hooked. I gather this experience is similar to others who are now Python advocates, who are using Python for their personal projects, etc. In the short term, though, for the corporate world, it'll probably be an uphill battle to have major projects developed in Python (or even Jython, which would be the logical choice for a transition out of j2ee to the lamp world). There is some amount of resistance to open source software. The same resistance to linux was overcome by 'respectable' companies supporting and interacting on a corporate level, like Red Hat. IBM certainly helped there, too, like it or not. Python would probably do well to have similar corporate advocates, but that would be an expensive and risky business proposition. Short term, I wouldn't expect to be able to make a big paycheck out of Python. As time goes on and the younger programmers who are using Python on their personal and open source projects get into positions of influence in the corporate world that will change. My thoughts, for what it's worth. Fight the good fight! Michael On Fri, 2007-10-26 at 04:28 +0200, [EMAIL PROTECTED] wrote: > Send Python-Dev mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/python-dev > or, via email, send a message with subject or body 'help' to > [EMAIL PROTECTED] > > You can reach the person managing the list at > [EMAIL PROTECTED] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-Dev digest..." > > > Today's Topics: > >1. Re: Python developers are in demand (Nick Efford) >2. Re: Python developers are in demand (Facundo Batista) >3. Re: Python tickets summary (Facundo Batista) >4. Re: Python developers are in demand (Anthony Roy) >5. Re: Python developers are in demand > (Kevin Jacobs <[EMAIL PROTECTED]>) >6. Re: Python developers are in demand (Anna Ravenscroft) >7. Re: Python developers are in demand (Anna Ravenscroft) >8. Re: Python developers are in demand (Titus Brown) >9. Re: Python tickets summary (Ron Adam) > 10. Re: Fwd: Deadlock by a second import in a thread (Facundo Batista) > > > -- > > Message: 1 > Date: Thu, 25 Oct 2007 11:58:46 +0100 > From: Nick Efford <[EMAIL PROTECTED]> > Subject: Re: [Python-Dev] Python developers are in demand > To: [email protected] > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Interesting to see discussion on supply and demand issues for > Python programmers. You might be interested to learn that, > after a few years of flirting with Python in various ways, the > School of Computing at the University of Leeds has recently > switched to teaching Python as the first and primary programming > language for undergraduates on all of our degree programmes. > > I know we're not the only ones doing this, so perhaps the > supply will rise to meet the demand in a few years! > > > Nick > > -- > Dr Nick Efford, School of | E: [EMAIL PROTECTED] > Computing, University of | T: +44 113 343 6809 > Leeds, Leeds, LS2 9JT, UK | W: http://www.comp.leeds.ac.uk/nde/ > --+- > PGP fingerprint: 6ADF 16C2 4E2D 320B F537 8F3C 402D 1C78 A668 8492 > > > > -- > > Message: 2 > Date: Thu, 25 Oct 2007 09:13:48 -0300 > From: "Facundo Batista" <[EMAIL PROTECTED]> > Subject: Re: [Python-Dev] Python developers are in demand > To: "Alex Martelli" <[EMAIL PROTECTED]> > Cc: " Martin v. L?wis " <[EMAIL PROTECTED]>, Christian Heimes > <[EMAIL PROTECTED]>, Python Dev > Message-ID: > <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=ISO-8859-1 > > 2007/10/24, Alex Martelli <[EMAIL PROTECTED]>: > > > using C++ and Java (and often C), but as far as I know there is no > > Stanford course (at least not within Symbolic Systems) that focuses > > specifically and exclusively on Python (there IS one course, > > In my constant try-to-push-Python-everywhere-I-go, I offered several > times Python courses to educational institutions (sometimes even > free). > > I succeeded some times, but then these courses not th
Re: [Python-Dev] Does Python need a file locking module (slightly higher level)?
Barry> I don't think any one solution will work for everybody. I'm not Barry> even sure we can define a common API a la the DBAPI, but if Barry> something were to make it into the standard distribution, that's Barry> the direction I'd go in. I've been working on a lockfile module the past few days on the train. I have something that passes a boatload of doctest test cases on my Mac, works for threads as well as processes (acquire the lock in thread one, then block in thread two until thread one releases the lock). The Unix version relies on the atomic nature of the link(2) system call. The Windows version (not yet tested on that platform) relies on mkdir(2) being atomic. (I think it is.) In theory, I suppose the mkdir version will work for both platforms, so it's possible that you could have file locking between Windows and Unix should you want it. The current implementation provides a FileLock class with these methods: acquire release is_locked break_lock __enter__ __exit__ The acquire method takes an optional timeout parameter (None => wait forever, +ive value => block for awhile, zero or -ive => give up immediately). The others all take no arguments. I'm working on ReST documentation now and hope to have that finished over the weekend. After that I'll write a simple setup.py, toss it out on PyPI, then announce it more broadly. If anyone would like to try it out and/or review the code sooner (especially if you have access to Windows) let me know. I'll shoot you a copy as it currently exists. The API and almost all test cases are defined in a _FileLock base class. You could (in theory at least) subclass it to provide locking through some other shared resource like a database and not have to write and or many other test cases. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deadlock by a second import in a thread
On 10/25/07, Facundo Batista <[EMAIL PROTECTED]> wrote: > 2007/10/25, Facundo Batista <[EMAIL PROTECTED]>: > > > BTW, I'll leave the optimization of importing strptime one time, > > there's no reason to try to import it everytime strptime() is called. > > No, I'm not. In consideration to the possible warning raised by Brett, > I won't commit the change (it does not fix anything, just a marginal > optimization, so there's no enough reason to bring a possible issue). > > For the record, I'm attaching the would-be patch. > I honestly have no problem if you do the import once, Facundo. My warning is so marginal I wouldn't worry about it. Plus almost all other code, C and Python alike, assume the import happens only once at import so it shouldn't be a surprise to anyone. > Thank you all! Thanks for putting the time in to figure this out! -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] resurrected modindex
A long time ago I wrote a redirector to note and display the most frequently requested modules from the library reference manual's global module index. When Georg Brandl did his massive rewrite of the Python manuals that broke. I finally took the time this evening to resurrect that function. The result is here: http://www.webfast.com/modindex/ I reset all the counters and seeded the cache with a few modules. Click around in the index a few times then reload the above page. You'll get the idea. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
