pysqlite throwing exception?
I'm trying to modify an app I wrote a few months ago, but now it dies on startup (it worked before). The app loads the SQLite Media Monkey database, and crashes on its first query (when I try to get the number of podcasts). At the end of this post is a reduced version of the problem (which produces the same error). Oh, this is Python 2.5 on Windows. The following is the traceback for the reduced version: Traceback (most recent call last): File C:\projects\podcasts\podstrand-mm\temp.py, line 16, in module cursor.execute(select count(*) as count from songs) OperationalError: malformed database schema - near VIRTUAL: syntax error It's the same error I get while running this code in context. The major change is that I upgraded version of MediaMonkey, and I'd think that might cause problems-- although all of my non-Python SQLite support apps have no problem (including running the same query). Unfortunately, if pysqlite's been upgraded, I can't find it -- pysqlite.org has been down the past 2 days. -Wm import os import sqlite3 # Find the mediamonkey database. conn = sqlite3.connect( os.path.join(os.environ['USERPROFILE'], 'Local Settings', 'Application Data', 'MediaMonkey', 'MM.DB') ) conn.row_factory = sqlite3.Row # provide named columns in results # Ask mediamonkey for its data. cursor = conn.cursor() # Get the total number of songs. cursor.execute(select count(*) as count from songs) track_estimate = cursor.fetchall()[0]['count'] -- http://mail.python.org/mailman/listinfo/python-list
Re: pysqlite throwing exception?
william tanksley wtanksle...@gmail.com wrote: Oh, this is Python 2.5 on Windows. New result: this works on Python 2.6. Obviously the SQLite format changed between the two runs. I'll call this problem solved; my app appears to run now. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: pysqlite throwing exception?
MRAB pyt...@mrabarnett.plus.com wrote: I wonder whether it's complaining about the as count part because count is the name of a function, although you do say that the same query works elsewhere. Hey, good catch. Thanks; I'll change that. (It wasn't the problem, but no doubt someday it could be.) -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido's new method definition idea
On Dec 5, 6:21 pm, Daniel Fetchinson [EMAIL PROTECTED] wrote: I'd like this new way of defining methods, what do you guys think? Anyone ready for writing a PEP? I think it's an awesome proposal. It's about time! With this change, defining methods uses the same special syntax hack that calling them does. This is a good thing because it makes it easy to refer to methods consistently. I see a lot of people are against it; I admit that it's not the status quo, but that's not a sufficient argument against a change (it defeats all possible changes). A more interesting argument against it is that it's special implicit syntax; but I would argue that it merely reflects the existing special syntax of method calls. Unfortunately, writing a PEP is sadly outside my skillset. Daniel -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: RegExp: wontmatch-function
On Oct 13, 9:40 am, [EMAIL PROTECTED] wrote: I'm looking for a function which, given a regexp re and and a string str, returns whether re won't match any string starting with str. (so it would always return False if str is or if str itself matches re -- but that are only the easy cases). Wow, that's a pretty problem. Any ideas or prior art on how to get this function? An idea, which I haven't explored well enough: Parse the regexp, and recursively descend the parse tree; for each node on the recursive descent, generate the regexp that follows the descent to end at that node and ask if it matches the string. If it matches, return 'False'. If the recursion ends without ever matching, return 'True'. There are a few questions on how to do that. I don't know how to parse the regexp (but I know how to start and am confident that it'd be easy); I don't know how to generate the regexp given a descent path (but I think I could do it). I'm also not sure this is correct, but it's not a homework assignment for me, so I'm not worried. Peter -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Bidirectional Generators
Paddy [EMAIL PROTECTED] wrote: What's one of them then? I'm sorry, I don't know what you mean. Meanwhile, more pertinently: I did get my generator working, and then I replaced it with a class that did the same thing in less than a quarter of the number of lines. So... I'm not going to worry about that anymore. My use case obviously wasn't the right one for them. I'm still curious, though, whether anyone's written any code that actually uses yield _and_ send() to do anything that isn't in the original PEP. - Paddy. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Bidirectional Generators
Jeff [EMAIL PROTECTED] wrote:
william tanksley [EMAIL PROTECTED] wrote:
I'm still curious, though, whether anyone's written any code that
actually uses yield _and_ send() to do anything that isn't in the
original PEP.
I have. An iterator that could backtrack itself without the user
having to remember previous states. It would just send back something
like reader.send('prev_token') or reader.send(-1).
Ah, nice! I now remember briefly thinking that such a thing should be
possible. Interestingly, I just recently wrote a backtracking search,
so I could have used such a thing -- it's a pity that I didn't
remember about send() while I was doing that.
I may have to alter my design to see what happens.
-Wm
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
John Machin [EMAIL PROTECTED] wrote: william tanksley [EMAIL PROTECTED] wrote: Cool. Sorry for the misunderstanding. Thank you for helping again! Postscript: your request to print the actual data did the trick. I'd back inspecting actual data against armchair philosophy any time :-) Heh. It's a recurring problem with me, to tell the truth. You're right that single quotes are expected -- and I'd expect a preceding u, since they're supposed to be Unicode. I dunno what's going on. Why do you suppose that the contents are Unicode? It's a URL-encoded string i.e. *deliberately* ASCII, in fact sub-ASCII (see all the %20 stuff?). What's going on is that ElementTree presents text as ASCII if it can be so represented, otherwise as Unicode. This is actually a *convenience*. Get used to it. Enjoy it. This isn't what caused the problem, but how is it convenient to get Unicode sometimes and ASCII other times? Given that the input file was Unicode, and in fact some of the values required Unicode, I'd expect to have gotten Unicode out for everything. I don't see how it matters; as far as I know, the methods available for Unicode and ASCII strings are the same, and only the type() is different. So I'm not saying it's a problem; I'm just not seeing how it's a _convenience_. So all of the mysteries are solved (except for my Python's doublequotes, but who cares), and ElementTree is entirely vindicated. Shucks. I can sense that you'd been looking forward to conducting an auto-da-fe followed by tossing the author on a bonfire ... but you can't burn a bot anyway :-) Well, I really _was_ expecting the Spanish Inquisition. Darn. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
Stefan Behnel [EMAIL PROTECTED] wrote: william tanksley wrote: Okay, my answer is that ElementTree (in Python 2.5) is simply deranged when it comes to Unicode. It assumes everything's ASCII. It does not assume that. It *requires* byte strings to be ASCII. You can't encode Unicode into an ASCII string. (Well, except using UTF-7.) Bad requirement. If it didn't enforce that, how could it possibly know what encoding they were using, i.e. what they were supposed to mean at all? Read the Python Zen, in the face of ambiguity, ElementTree refuses the temptation to guess. Python 2.x does exactly the same thing when it comes to implicit conversion between encoded strings and Unicode strings. An XML file that begins with the string ?xml encoding=utf-8? is NOT ascii. You don't have to guess what encoding it's in. It's UTF-8. If you error out when you hit an 8-bit character, you're not going to be able to process that file. I'm completely lost on why you're claiming otherwise. Furthermore, when ElementTree returns (from one of its .text elements) a string-of-bytes instead of a decoded Unicode string, it doesn't merely resist the temptation to guess; instead, it forces ME to guess. I've now had to hardcode utf-8 into my program, when IT just bypassed and ignored an explicit instruction to use UTF-8. I hope and assume that iTunes will never switch from UTF-8 to UTF-32 -- if it does, my code breaks, and I'll probably have to switch away from ElementTree (I guess that since it requires ASCII it won't even pretend to handle more than 8 bits per character). If you want to pass plain ASCII strings, you can either pass a byte string or a Unicode string (that's a plain convenience feature). If you want to pass anything that's not ASCII, you *must* pass a Unicode string. I don't care about strings. I've never passed ElementTree a string. I'm using a file, a file that's correctly encoded as UTF-8, and it returns some text elements that are raw bytes (undecoded). I have to manually decode them. Reference:http://codespeak.net/lxml/compatibility.html (Note that the lxml version also doesn't handle Unicode correctly; it errors when XML declares its encoding.) It definitely does handle Unicode correctly. Actually, this is my bad -- I misread the webpage. lxml appears to handle unicode strings with a declared encoding correctly: it errors out. That's quite reasonable when confronted with a contradiction. According to that page, however, the standard ElementTree library doesn't work that way -- it simply assumes that byte strings are ASCII. I'm going to back down on this one, though. I realize that this is a single paragraph on a third-party website, and it's not really trying to document the official ElementTree (it's trying to document its own version, lxml). So it might not be correct, or it might be overly ambiguous. It might also be talking ONLY about strings, to the exclusion of file input. I don't know, and I don't have the energy to debug it, especially since I can't fix anything about it even if something was wrong :-). So I revert to my former position: I don't know why those two lines have to be in that order for my code to work correctly; I don't even know why the encode line has to be there at all. When I was using the old Python XML library, I didn't have to worry about encoding or decoding; everything just worked. I really prefer ElementTree, and I'm glad I upgraded, but it really looks like encoding is a problem. Let me guess, you tried passing XML as a Unicode string into the parser, and your XML declared itself as having a byte encoding (?xml encoding=...?). How can that *not* be an error? I thought you just said resist the temptation to guess? I didn't pass a string. I passed a file. It didn't error out; instead, it produced bytestring-encoded output (not Unicode). Stefan -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
John Machin [EMAIL PROTECTED] wrote: william tanksley [EMAIL PROTECTED] wrote: Buffett Time - Annual Shareholders\xc2\xa0L.mp3 1. This isn't Unicode; it's missing the u (I printed using repr). 2. It's got the UTF-8 bytes there in the middle. In addition to the above results, *WHAT* results? I don't see any repr() output, just your interpretation of what you think you saw! That *is* the repr. I said it's the repr, and it IS. It's not an interpretation; it's a screenscrape. Really, truly. If I paste it in again it'll look the same. What do you want? Can I post something that will convince you it's a repr? Oh well. You guys have been immensely helpful; my mental model of how Python works was vastly backwards, so it's a relief to get it corrected. Thanks to that, I was able to hack my code into working. I wish I could get entirely correct behavior, but at this point the miscommunication is too strong. I'll settle for the hack I've got now, and hope iTunes doesn't ever change its XML encoding (hey, I think I've got cause to be optimistic). -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
John Machin [EMAIL PROTECTED] wrote: william tanksley [EMAIL PROTECTED] wrote: Let's try again: Cool. Sorry for the misunderstanding. Thank you for helping again! Postscript: your request to print the actual data did the trick. I'm including the rest of my reply just to provide context, but the answer was the the Unicode was actually embedded in the URL, encoded as distinct bytes. Thus, it *had* to be url-decoded and then UTF-8 decoded, in that order, in order to recover the original filename. So the problem was indeed purely in my head -- I should have looked at the original data (unfortunately, I was fooled by looking at the song title, which is the same thing but with the raw UTF-8 bytes instead of the URL escape codes). track_id = url2pathname(urlparse(track_id).path) print repr(track_id) parse_result = urlparse(track_id).path print repr(parse_result) track_id_replacement = url2pathname(parse_result) print repr(track_id_replacement) The important value here is track_id_replacement; it contains the data that's throwing me. It appears that some UTF-8 characters are being read as multiple bytes by ElementTree rather than being decoded into Unicode. Here's one example. The others are similar -- they have the same things that look like problems to me. Buffett Time - Annual Shareholders\xc2\xa0L.mp3 ROTFL! I thought the Buffett thing was a Windows filename! What I was expecting was THREE lots of repr() output, and I'm quite unused to seeing repr() output with quotes around it instead of apostrophes; how did you achieve that? I don't know -- but I got it again when I printed out the original version. My *guess* would be that this is what repr prints when asked to print a byte string (but I don't know how to confirm that). Alternately, the fact that I'm running these inside SPE might be changing some defaults. I'm not sure. You're right that single quotes are expected -- and I'd expect a preceding u, since they're supposed to be Unicode. I dunno what's going on. So you're saying that track_id_replacement contains utf8 characters. It is obtained by track_id_replacement = url2pathname(parse_result). You don't show us what is in parse_result. url2pathname() is nothing to do with ElementTree. urlparse() is nothing to do with ElementTree. You have provided no evidence that ElementTree is doing what you accuse it of. Okay. Here's the evidence... Or something. Looking at this I begin to see why things work the way they do. It's utterly bizzare, quite frankly. Please try again. Backtrack in your code to where you are pulling the url out of an element. Do print repr(some_element.some_attribute). Show us. Okay, the repr of the string that comes out of the .text attribute is: file://localhost/C:/Documents%20and%20Settings/TanksleyJrW/My %20Documents/My%20Music/iTunes/iTunes%20Music/Podcasts/Brian %20Preston's%20_Money%20Guy_%20Blog%20and%20Pod/Buffett%20Time%20- %20Annual%20Shareholders%C2%A0L.mp3 Looking at the XML, and THIS TIME actually looking at the correct attribute (I was looking at the title before) I see... surprise! That's the correct data. So all of the mysteries are solved (except for my Python's doublequotes, but who cares), and ElementTree is entirely vindicated. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
Thank you for the response. Here's some more info, including a little that you didn't ask me for but which might be useful. John Machin [EMAIL PROTECTED] wrote: william tanksley [EMAIL PROTECTED] wrote: To ask another way: how do I convert from a file:// URL to a local path in a standard way, so that filepaths from two different sources will work the same way in a dictionary? The problems occur when the filenames have non-ascii characters in them -- I suspect that the URLs are having some encoding placed on them that Python's decoder doesn't know about. # track_id = url2pathname(urlparse(track_id).path) print repr(track_id) parse_result = urlparse(track_id).path print repr(parse_result) track_id_replacement = url2pathname(parse_result) print repr(track_id_replacement) The important value here is track_id_replacement; it contains the data that's throwing me. It appears that some UTF-8 characters are being read as multiple bytes by ElementTree rather than being decoded into Unicode. Could this be a bug in ElementTree's Unicode support? If so, can I work around it? Here's one example. The others are similar -- they have the same things that look like problems to me. Buffett Time - Annual Shareholders\xc2\xa0L.mp3 Note some problems here: 1. This isn't Unicode; it's missing the u (I printed using repr). 2. It's got the UTF-8 bytes there in the middle. I tried doing track_id.encode(utf-8), but it doesn't seem to make any difference at all. Of course, my ultimate goal is to compare the track_id to the track_id I get from iTunes' COM interface, including hashing to the same value for dict lookups. and copy/paste the results into your next posting. In addition to the above results, while trying to get more diagnostic printouts I got the following warning from Python: C:\projects\podcasts\podstrand\podcast.py:280: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal return track.databaseID == trackLocation The code that triggered this is as follows: if trackLocation in self.podcasts: track = self.podcasts[trackLocation] if trackRelease: track.release_date = trackRelease elif track.is_podcast: print No release date:, repr(track.name) else: # For the sake of diagnostics, try to find the track. def track_has_location(track): return track.databaseID == trackLocation fillers = filter(track_has_location, self.fillers) if len(fillers): return disabled = filter(track_has_location, self.deferred) if len(disabled): return print Location not known:, repr(trackLocation) -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
Jerry Hill [EMAIL PROTECTED] wrote:
william tanksley [EMAIL PROTECTED] wrote:
Here's one example. The others are similar -- they have the same
things that look like problems to me.
Buffett Time - Annual Shareholders\xc2\xa0L.mp3
I tried doing track_id.encode(utf-8), but it doesn't seem to make
any difference at all.
I don't have anything to say about your iTunes problems, but encode()
is the wrong method to turn a byte string into a unicode string.
Instead, use decode(), like this:
Awesome... Thank you! I had my mental model of Python turned around
backwards. That's an odd feeling. Okay, so you decode to go from raw
byes into a given encoding, and you encode to go from a given encoding
to raw bytes. Not what I thought it was, but that's cool, makes sense.
At first I thought this fixed my problem, but I had to tweak the
obvious fix to make it work, and I don't understand why.
Fix #1:
track_id = track_id.decode('utf-8')
track_id = url2pathname(urlparse(track_id).path)
That doesn't work -- it produces no error, but the raw bytes appear in
the unicode string.
Fix #2:
track_id = url2pathname(urlparse(track_id).path)
track_id = track_id.decode('utf-8')
This one appears to work. (Although I can't confirm it for sure,
because although all my debug prints are now correct, the overall
application fails in the same way it did before, back before I put in
debug printfs. I'm going to spend some time assuming that the problem
is elsewhere in my code, since at least I definitely fixed one serious
problem.)
I've got a few questions for Python-XML-Unicode experts...
1. Why does the order of those statements matter?
2. Shouldn't it be more correct to decode BEFORE transforming the
string? Why does that kill the decoding?
3. Why is ElementTree dumping raw bytes on me instead of decoding to
UTF-8? The XML file has its encoding set to:
?xml version=1.0 encoding=UTF-8?, so it seems like it should
know what codec to use.
Jerry
-Wm
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
Jerry Hill [EMAIL PROTECTED] wrote: On Wed, Jul 30, 2008 at 2:27 PM, william tanksley [EMAIL PROTECTED] wrote: Awesome... Thank you! I had my mental model of Python turned around backwards. That's an odd feeling. Okay, so you decode to go from raw byes into a given encoding, and you encode to go from a given encoding to raw bytes. Not what I thought it was, but that's cool, makes sense. That's not quite right. Decoding takes a byte string that is already in a particular encoding and transforms it to unicode. Unicode isn't a encoding of it's own. Decoding takes a unicode string (which doesn't have any encoding associated with it), and gives you back a sequence of bytes in a particular encoding. Okay, this is useful. Thank you for straightening out my mental model. It makes sense to define strings as just naturally Unicode... and anything else is in some ways not really a string, although it's something that might have many of the same methods. I guess this mental model is being implemented more thoroughly in Py3K... Anyhow, it makes sense. I'm still puzzled why I'm getting some non-Unicode out of an ElementTree's text, though. Jerry -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
william tanksley [EMAIL PROTECTED] wrote: I'm still puzzled why I'm getting some non-Unicode out of an ElementTree's text, though. Now I know. Okay, my answer is that cElementTree (in Python 2.5) is simply deranged when it comes to Unicode. It assumes everything's ASCII. Reference: http://codespeak.net/lxml/compatibility.html (Note that the lxml version also doesn't handle Unicode correctly; it errors when XML declares its encoding.) This is unpleasant, but at least now I know WHY it was driving me insane. -Wm -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: Python parsing iTunes XML/COM
To ask another way: how do I convert from a file:// URL to a local path in a standard way, so that filepaths from two different sources will work the same way in a dictionary? Right now I'm using the following source: track_id = url2pathname(urlparse(track_id).path) url2pathname is from urllib; urlparse is from the urlparse module. The problems occur when the filenames have non-ascii characters in them -- I suspect that the URLs are having some encoding placed on them that Python's decoder doesn't know about. Thank you all in advance, and thank you for Python. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: iterating by twos
kj [EMAIL PROTECTED] wrote: Is there a special pythonic idiom for iterating over a list (or tuple) two elements at a time? I don't know of one, and I shouldn't be answering, but the following should work: def gulp_two(items): for i,j in zip(items[0::2], items[1::2]): yield (i,j) Let's see... list(gulp_two(range(10))) [(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)] Okay, it works. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Python parsing iTunes XML/COM
I'm trying to convert the URLs contained in iTunes' XML file into a form comparable with the filenames returned by iTunes' COM interface. I'm writing a podcast sorter in Python; I'm using iTunes under Windows right now. iTunes' COM provides most of my data input and all of my mp3/aac editing capabilities; the one thing I can't access through COM is the Release Date, which is my primary sorting field. So I read everything in through COM, then read all the release dates from the iTunes XML file, then try to join the two together... But so far I have zero success. Is there _any_ way to match up tracks between iTunes COM and iTunes XML? I've spent far too much effort on this. I'm not stuck on using filenames, if that's a bad idea... But I haven't found anything else that works, and filenames seem like an obvious solution. -Wm -- http://mail.python.org/mailman/listinfo/python-list
Bidirectional Generators
Okay, I'm almost finished with my first bidirectional generator. By almost finished I mean both that it's almost working, and that I'm almost about to replace it with a class that works a bit more like what I currently understand. Surely some other people have worked with this feature... Are there any pages that discuss how it's been useful? No, I don't want to see an implementation of coroutines. I get that one already. :-) -Wm -- http://mail.python.org/mailman/listinfo/python-list
Re: How to get an XML DOM while offline?
Diez B. Roggisch [EMAIL PROTECTED] wrote: The most pragmatic solution would be to rip the doctype out using simple string methods and/or regexes. Thank you, Diez and Paul; I took Diez's solution, and it works well enough for me. Diez -Wm -- http://mail.python.org/mailman/listinfo/python-list
How to get an XML DOM while offline?
I want to parse my iTunes Library xml. All was well, until I unplugged and left for the train (where I get most of my personal projects done). All of a sudden, I discovered that apparently the presence of a DOCTYPE in the iTunes XML makes xml.dom.minidom insist on accessing the Internet... So suddenly I was unable to do any work. I don't want to modify the iTunes XML; iTunes rewrites it too often. How can I prevent xml.dom.minidom from dying when it can't access the Internet? Is there a simpler way to read the iTunes XML? (It's merely a plist, so the format is much simpler than general XML.) -Wm -- http://mail.python.org/mailman/listinfo/python-list
