[issue25472] Typing: Specialized subclasses of generics cannot be unpickled
New submission from Matt Chaput:
If I try to pickle and unpickle an object of a class that has specialized a
generic superclass, when I try to unpickle I get this error:
TypeError: descriptor '__dict__' for 'A' objects doesn't apply to 'B' object
Test case:
from typing import Generic, TypeVar
import pickle
T = TypeVar("T")
class A(Generic[T]):
def __init__(self, x: T):
self.x = x
class B(A[str]):
def __init__(self, x: str):
self.x = x
b = B("hello")
z = pickle.dumps(b)
print(z)
_ = pickle.loads(z)
--
messages: 253421
nosy: maatt
priority: normal
severity: normal
status: open
title: Typing: Specialized subclasses of generics cannot be unpickled
versions: Python 3.5
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25472>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6623] Lib/ftplib.py Netrc class should be removed.
Matt Chaput added the comment: This patch is the same as my previous one, except instead of removing Netrc usage from the ftplib.test() function, it replaces it with the netrc.netrc object. Note that there are no existing tests for the ftplib.test() function. Also did some very minor cleanups (bare raise is no longer valid) to get rid of warnings/errors in static analyzer. -- Added file: http://bugs.python.org/file34818/remove_Netrc_class2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6623 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21198] Minor tarfile documentation bug
Matt Chaput added the comment: Simple patch to remove the underscore in tarfile.rst. -- keywords: +patch nosy: +maatt Added file: http://bugs.python.org/file34824/issue21198.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21198 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21146] update gzip usage examples in docs
Matt Chaput added the comment: The patch looks good to me. -- nosy: +maatt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21146 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20491] textwrap: Non-breaking space not honored
Matt Chaput added the comment: Patch on top of dbudinova's that attempts to replace the concatenation of strings with a verbose regex. -- nosy: +maatt Added file: http://bugs.python.org/file34827/issue20491_verbose.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20491 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21198] Minor tarfile documentation bug
Changes by Matt Chaput m...@whoosh.ca: Removed file: http://bugs.python.org/file34824/issue21198.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21198 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21198] Minor tarfile documentation bug
Matt Chaput added the comment: Oops! Yes, I accidentally included a bunch of other crap. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue21198 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6623] Lib/ftplib.py Netrc class should be removed.
Matt Chaput added the comment: Created patch to remove the Netrc class and its unit tests (for Python 3.5). -- nosy: +maatt Added file: http://bugs.python.org/file34806/remove_Netrc_class.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6623 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput added the comment: IIRC the root issue turned out to be that when you execute any multiprocessing statements at the module/script level on Windows, you need to put it under if __name__ == __main__, otherwise it will cause infinite spawning. I think this is mentioned in the multiprocessing docs but should probably be in giant blinking red letters ;) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14447] marshal.load() reads entire remaining file instead of just next value
New submission from Matt Chaput m...@whoosh.ca:
In Python 3.2, if you write several values to a file with multiple calls to
marshal.dump(), and then try to read them back, the first marshal.load()
returns the first value, but reads to the end of the file, so subsequent calls
to marshal.load() raise an EOFError.
E.g.:
import marshal
f = open(test, wb)
marshal.dump((hello, 1), f)
marshal.dump((there, 2), f)
marshal.dump((friend, 3), f)
f.close()
f = open(test, rb)
print(marshal.load(f)) # ('hello', 1)
print(marshal.load(f)) # ERROR
This page seems to indicate this was also a bug in Python 3.1:
http://www.velocityreviews.com/forums/t728526-python-3-1-2-and-marshal.html
--
components: IO
messages: 157093
nosy: mattchaput
priority: normal
severity: normal
status: open
title: marshal.load() reads entire remaining file instead of just next value
type: behavior
versions: Python 3.1, Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14447
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Spamming PyPI with stupid packages
Someone seems to be spamming PyPI by uploading multiple stupid packages. Not sure if it's some form of advertising spam or just idiocy. Don't know if we should care though... maybe policing uploads is worse than cluttering PyPI's disk space and RSS feed with dumb 1 KB packages. girlfriend 1.0.1 10 A really simple module that allow everyone to do import girlfriend girlfriends 1.0 4 Girl Friends car 1.0 2 Car, a depended simple module that allow everyone to do import girlfriend house 1.0 2 House, a depended simple module that allow everyone to do import girlfriend money 1.0 2 Money, a depended simple module that allow everyone to do import girlfriend workhard 1.0 2 Keep working hard, a depended simple module that allow everyone to do import girlfriend Matt -- http://mail.python.org/mailman/listinfo/python-list
[issue12986] Using getrandbits() in uuid.uuid4() is faster and more readable
Matt Chaput m...@whoosh.ca added the comment: Passed all tests OK. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12986 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12986] Using getrandbits() in uuid.uuid4() is faster and more readable
New submission from Matt Chaput m...@whoosh.ca: Currently the 'uuid' module uses os.urandom (in the absence of a system UUID generation function) to generate random UUIDs in the uuid.uudi4() function. This patch changes the implementation of uuid4() to use random.getrandbits() as the source of randomness instead, for the following reasons: * In my quick tests, using getrandbits() is much faster on Windows and Linux. Some applications do need to generate UUIDs quickly. setup = import uuid, os, random ur = uuid.UUID(bytes=os.urandom(16), version=4) grb = uuid.UUID(int=random.getrandbits(128), version=4) # Windows timeit.Timer(ur, setup).timeit() 22.861042160383903 timeit.Timer(grb, setup).timeit() 3.8689128309085135 # Linux timeit.Timer(ur, setup).timeit() 29.32686185836792 timeit.Timer(grb, setup).timeit() 3.7429409027099609 * The patched code is cleaner. It avoids the try...finally required by the possibly unavailable os.urandom function, and the fallback to generating random bytes. -- components: Library (Lib) files: fastuuid4.patch keywords: patch messages: 144087 nosy: mattchaput priority: normal severity: normal status: open title: Using getrandbits() in uuid.uuid4() is faster and more readable type: performance Added file: http://bugs.python.org/file23163/fastuuid4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12986 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2636] Adding a new regex module (compatible with re)
Matt Chaput m...@whoosh.ca added the comment: Not sure if this is better as a separate feature request or a comment here, but... the new version of .NET includes an option to specify a time limit on evaluation of regexes (not sure if this is a feature in other regex libs). This would be useful especially when you're executing regexes configured by the user and you don't know if/when they might go exponential. Something like this maybe: # Raises an re.Timeout if not complete within 60 seconds match = myregex.match(mystring, maxseconds=60.0) -- nosy: +mattchaput ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue2636 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12870] Regex object should have introspection methods
Matt Chaput m...@whoosh.ca added the comment: Yes, it's an optimization of my code, not the regex, as I said. Believe me, it's not premature. I've listed two general use cases for the two methods. To me it seems obvious that having to test a large number of regexes against a string, and having to test a single regex against a large number of strings, are two very common programming tasks, and they could both be speeded up quite a bit using these methods. As of now my parsing code and other code such as PyParsing are resorting to hacks like requiring the user to manually specify the possible first chars of a regex at configuration. With the hacks, the code can be hundreds of times faster. But the hacks are error-prone and should be unnecessary. The PCRE library implements at least the first char functionality, and a lot more regex introspection that would be useful, through its pcre_fullinfo() function. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12870 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12870] Regex object should have introspection methods
Matt Chaput m...@whoosh.ca added the comment: Ezio, no offense, but I think it's safe to say you've completely misunderstood this bug. It is not about explaining what a regex matches or optimizing the regex. Read the last sentences of the two paragraphs explaining the proposed methods for the use cases. This is about allowing MY CODE to programmatically get certain information about a regex object to allow it to limit the number of times it has to call regex.match(). AFAIK there's no good way to get this information about a regex object without adding these methods or building my own pure-Python regex interpreter, which would be both Herculean and pointless. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12870 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12870] Regex object should have introspection methods
New submission from Matt Chaput m...@whoosh.ca: Several times in the recent past I've wished for the following methods on the regular expression object. These would allow me to speed up search and parsing code, by limiting the number of regex matches I need to try. literal_prefix(): Returns any literal string at the start of the pattern (before any special parts). E.g., for the pattern ab(c|d)ef the method would return ab. For the pattern abc|def the method would return . When matching a regex against keys in a btree, this would let me limit the search to just the range of keys with the prefix. first_chars(): Returns a string/list/set/whatever of the possible first characters that could appear at the start of a matching string. E.g. for the pattern ab(c|d)ef the method would return a. For the pattern [a-d]ef the method would return abcd. When parsing a string with regexes, this would let me only have to test the regexes that could match at the current character. As long as you're making a new regex package, I thought I'd put in a request for these :) -- components: Regular Expressions messages: 143266 nosy: mattchaput priority: normal severity: normal status: open title: Regex object should have introspection methods type: feature request versions: Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12870 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: Looking for Coders or Testers for an Open Source File Organizer
On 13/06/2011 11:55 PM, zainul franciscus wrote: Iknow you guys must be thinking Hmm, Miranda, isn't that an IM application ?; Yep I hear you, I'll change the name once I get a good name. I am open for any suggestions. Actually I was thinking isn't that a functional programming language? My suggestion: Cruftbuster -- http://mail.python.org/mailman/listinfo/python-list
Snowball to Python compiler
On the slim chance that (a) somebody worked on something like this but never uploaded it to PyPI, and (b) the person who did (a) or heard about it is reading this list ;) -- I'm looking for some code that will take a Snowball program and compile it into a Python script. Or, less ideally, a Snowball interpreter written in Python. (http://snowball.tartarus.org/) Anyone heard of such a thing? Thanks! Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Snowball to Python compiler
A third (more-than-) possible solution: google(python snowball); the first page of results has at least 3 hits referring to Python wrappers for Snowball. There are quite a few wrappers for the C-compiled snowball stemmers, but I'm looking for a pure-Python solution. It doesn't seem like there is such a thing, but I figured I'd take a shot here before I think about doing it myself :/ Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Tips on Speeding up Python Execution
On 08/04/2011 11:31 AM, Chris Angelico wrote: On Sat, Apr 9, 2011 at 12:41 AM, MRABpyt...@mrabarnett.plus.com wrote: On 08/04/2011 08:25, Chris Angelico wrote: [snip] I don't know what's the most Pythonesque option, but if you already have specific Python code for each of your functions, it's probably going to be easiest to spawn threads for them all. Pythonesque refers to Monty Python's Flying Circus. The word you want is Pythonic. And the word for referring to the actual snake is Pythonical :P -- http://mail.python.org/mailman/listinfo/python-list
Re: Reading/Writing files
On 18/03/2011 5:33 PM, Jon Herman wrote: I am pretty new to Python and am trying to write data to a file. However, I seem to be misunderstanding how to do so. For starters, I'm not even sure where Python is looking for these files or storing them. The directories I have added to my PYTHONPATH variable (where I import modules from succesfully) does not appear to be it. So my question is: How do I tell Python where to look for opening files, and where to store new files? This is how you write to a file in Python myfile = open(path/to/the/file, wb) myfile.write(Hello world!\n) myfile.close() Beyond that, your message is too vague to offer any real help, but it sounds like you're way off track. If the above code doesn't help, please tell us exactly what you're trying to do, but you might want to read a Python book such as Dive Into Python first. Cheers, Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Finding keywords
On 08/03/2011 8:58 AM, Cross wrote: I know meta tags contain keywords but they are not always reliable. I can parse xhtml to obtain keywords from meta tags; but how do I verify them. To obtain reliable keywords, I have to parse the plain text obtained from the URL. I think maybe what the OP is asking about is extracting key words from a text, i.e. a short list of words that characterize the text. This is an information retrieval problem, not really a Python problem. One simple way to do this is to calculate word frequency histograms for each document in your corpus, and then for a given document, select words that are frequent in that document but infrequent in the corpus as a whole. Whoosh does this. There are different ways of calculating the importance of words, and stemming and conflating synonyms can give you better results as well. A more sophisticated method uses part of speech tagging. See the Python Natural Language Toolkit (NLTK) and topia.termextract for more information. http://pypi.python.org/pypi/topia.termextract/ Yahoo has a web service for key word extraction: http://developer.yahoo.com/search/content/V1/termExtraction.html You might want to investigate these resources and try google searches for e.g. extracting key terms from documents and then come back if you have a question about the Python implementation. Cheers, Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Unit testing multiprocessing code on Windows
On 18/02/2011 2:54 AM, Terry Reedy wrote: On 2/17/2011 6:31 PM, Matt Chaput wrote: Does anyone know the right way to write a unit test for code that uses multiprocessing on Windows? I would start with Lib/test/test_multiprocessing. Good idea, but on the one hand it doesn't seem to be doing anything special, and on the other hand it seems to do it's own things like not having its test cases inherit from unittest.TestCase. I also don't know if the Python devs start it with distutils or nosetests, which are the ones I'm having a problem with. For example, starting my test suite inside PyDev doesn't show the bug. My test code isn't doing anything unusual... this is pretty much all I do to trigger the bug. (None of the imported code has anything to do with processes.) from __future__ import with_statement import unittest import random from whoosh import fields, query from whoosh.support.testing import TempIndex try: import multiprocessing except ImportError: multiprocessing = None if multiprocessing: class MPFCTask(multiprocessing.Process): def __init__(self, storage, indexname): multiprocessing.Process.__init__(self) self.storage = storage self.indexname = indexname def run(self): ix = self.storage.open_index(self.indexname) with ix.searcher() as s: r = s.search(query.Every(), sortedby=key, limit=None) result = .join([h[key] for h in r]) assert result == ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz, result class TestSorting(unittest.TestCase): def test_mp_fieldcache(self): if not multiprocessing: return schema = fields.Schema(key=fields.KEYWORD(stored=True)) with TempIndex(schema, mpfieldcache) as ix: domain = list(uabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ) random.shuffle(domain) w = ix.writer() for char in domain: w.add_document(key=char) w.commit() tasks = [MPFCTask(ix.storage, ix.indexname) for _ in xrange(4)] for task in tasks: task.start() for task in tasks: task.join() if __name__ == '__main__': unittest.main() -- http://mail.python.org/mailman/listinfo/python-list
Re: Unit testing multiprocessing code on Windows
On 17/02/2011 8:22 PM, phi...@semanchuk.com wrote: Hi Matt, I assume you're aware of this documentation, especially the item entitled Safe importing of main module? http://docs.python.org/release/2.6.6/library/multiprocessing.html#windows Yes, but the thing is my code isn't __main__, my unittest classes are being loaded by setup.py test or nosetests. And while I'm assured multiprocessing doesn't duplicate the original command line, what I get sure looks like it, because if I use python setup.py test that command seems to be re-run for every Process that starts, but if I use nosetests then *that* seems to be re-run for every Process. Matt -- http://mail.python.org/mailman/listinfo/python-list
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput m...@whoosh.ca added the comment: I don't know what to tell you... to the best of my knowledge there's absolutely no way for my code to kick off the entire test suite -- I always do that through PyDev (which doesn't cause the bug, by the way). The closest thing is the boilerplate at the bottom of every test file: if __name__ == __main__: unittest.main() ...but even that would only start the tests in that file, not the entire suite. Another thing that makes me think multiprocessing is re-running the original command line is that if I use python setup.py test to start the tests, when it gets to the MP tests it seems to run that command for each Process that gets started, but if I use nosetests, it seems to run nosetests for each started Process. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput m...@whoosh.ca added the comment: If I do c:\python27\python run_nose.py it works correctly. If I do nosetests I get the process explosion. Maybe the bug is in how distutils and nose work from the command line? I'm confused. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Unit testing multiprocessing code on Windows
Does anyone know the right way to write a unit test for code that uses multiprocessing on Windows? The problem is that with both python setup.py tests and nosetests, when they get to testing any code that starts Processes they spawn multiple copies of the testing suite (i.e. the new processes start running tests as if they were started with python setup.py tests/nosetests). The test runner in PyDev works properly. Maybe multiprocessing is starting new Windows processes by copying the command line of the current process? But if the command line is nosetests, it's a one way ticket to an infinite explosion of processes. Any thoughts? Thanks, Matt -- http://mail.python.org/mailman/listinfo/python-list
Unit testing multiprocessing code on Windows
Does anyone know the right way to write a unit test for code that uses multiprocessing on Windows? The problem is that with both python setup.py tests and nosetests, when they get to a multiprocessing test they spawn multiple copies of the testing suite. The test runner in PyDev works properly. Maybe multiprocessing is starting new Windows processes by copying the command line of the current process, but if the command line is nosetests, it's a one way ticket to an infinite explosion of processes. Any thoughts? Thanks, Matt -- http://mail.python.org/mailman/listinfo/python-list
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
New submission from Matt Chaput m...@whoosh.ca: If you start unit tests with a command line such as python setup.py test or nosetests, if the tested code starts a multiprocessing.Process on Windows, each new process will act as if it was started as python setup.py test/nosetests, leading to an infinite explosion of processes that eventually locks up the entire machine. -- components: Windows messages: 128768 nosy: mattchaput priority: normal severity: normal status: open title: Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11240] Running unit tests in a command line tool leads to infinite loop with multiprocessing on Windows
Matt Chaput m...@whoosh.ca added the comment: Thank you, I understand all that, but I don't think you understand the issue. My code is not __main__. I am not starting the test suite. It's the distutils/nose code that's doing that. It seems as if the multiprocessing module is starting new Windows processes by duplicating the command line of the original process. That doesn't seem to work very well, given the example of running test suites, hence the bug. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue11240 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Editor/IDE with Python coverage support?
Are there any editors/IDEs with good support for line-coloring from Python test coverage results? (I normally use Eclipse + PyDev but PyDev's current coverage support isn't much better than nothing.) Thanks, Matt -- http://mail.python.org/mailman/listinfo/python-list
[issue1172711] long long support for array module
Matt Chaput m...@sidefx.com added the comment: This is an important feature to me. Now I get to redo a bunch of code to have two completely different code paths to do the same thing because nobody could be bothered to keep array up-to-date. -- nosy: +mattchaput ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1172711 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Multiprocessing problem
Hi, I'm having a problem with the multiprocessing package. I'm trying to use a simple pattern where a supervisor object starts a bunch of worker processes, instantiating them with two queues (a job queue for tasks to complete and an results queue for the results). The supervisor puts all the jobs in the job queue, then join()s the workers, and then pulls all the completed results off the results queue. (I don't think I can just use something like Pool.imap_unordered for this because the workers need to be objects with state.) Here's a simplified example: http://pastie.org/850512 The problem is that seemingly randomly, but almost always, the worker processes will deadlock at some point and stop working before they complete. This will leave the whole program stalled forever. This seems more likely the more work each worker does (to the point where adding the time.sleep(0.01) as seen in the example code above guarantees it). The problem seems to occur on both Windows and Mac OS X. I've tried many random variations of the code (e.g. using JoinableQueue, calling cancel_join_thread() on one or both queues even though I have no idea what it does, etc.) but keep having the problem. Am I just using multiprocessing wrong? Is this a bug? Any advice? Thanks, Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing problem
On 3/2/2010 3:59 PM, Matt Chaput wrote: I'm trying to use a simple pattern where a supervisor object starts a bunch of worker processes, instantiating them with two queues (a job queue for tasks to complete and an results queue for the results). The supervisor puts all the jobs in the job queue, then join()s the workers, and then pulls all the completed results off the results queue. Here's a simplified example: http://pastie.org/850512 I should mention that if I change my code so the workers just pull things off the job queue but don't put any results on the result queue until after they see the None sentinel in the job queue and break out of the loop, I don't get the deadlock. So it's something about getting from one queue and putting to another queue in close proximity. Hopefully I'm making a simple mistake with how I'm using the library and it'll be easy to fix... Thanks, Matt -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing problem
If the main process doesn't get the results from the queue until the worker processes terminate, and the worker processes don't terminate until they've put their results in the queue, and the pipe consequently fills up, then deadlock can result. The queue never fills up... on platforms with qsize() I can see this. I remove items from the results queue as I add to the job queue, and if I add timeouts everywhere the workers never raise Empty and the supervisor never raises Full. They just deadlock. I've rewritten the code so the worker threads don't push information back while they run, they just write to a temporary file which the supervisor can read, which avoids the issue. But if anyone can tell me what I was doing wrong for future reference, I'd greatly appreciate it. Thanks, Matt -- http://mail.python.org/mailman/listinfo/python-list
[issue2027] Module containing C implementations of common text algorithms
Matt Chaput added the comment: The Porter stemming and Levenshtein edit-distance algorithms are not fast-moving nor are they fusion reactors... they've been around forever, and are simple to implement, but are still useful in various common scenarios. I'd say this is similar to Python including an implementation of digest functions such as SHA: it's useful enough, and compute-intensive enough, to warrant a C implementation. Shipping C extensions is not an option for everyone; it's especially a pain with Windows. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2027 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2027] Module containing C implementations of common text algorithms
New submission from Matt Chaput: Add a module to the standard library containing fast (C) implementations of common text/language related algorithms, to begin specifically Porter (and perhaps other) stemming and Levenshtein (and perhaps other) edit distance. Both these algorithms are useful in multiple domains, well known and understood, and have sample implementations all over the Web, but are compute-intensive and prohibitively expensive when implemented in pure Python. -- components: Library (Lib) messages: 62134 nosy: mchaput severity: normal status: open title: Module containing C implementations of common text algorithms type: rfe __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2027 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
