[issue36771] Feature Request: An option to os.walk() to return os.DirEntry lists instead of just filenames/dirnames
CJ Kucera added the comment: Will do, thanks for the input! -- ___ Python tracker <https://bugs.python.org/issue36771> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36771] Feature Request: An option to os.walk() to return os.DirEntry lists instead of just filenames/dirnames
CJ Kucera added the comment: Yeah, I'd wondered that too (re: a separate function) but it seemed like an awful lot of duplicated code. The PR I'd put through just changes the datatypes within the `filenames` and `dirnames` lists... I'd been thinking that'd be sufficient since you wouldn't get that without specifying the optional boolean, but perhaps not. -- ___ Python tracker <https://bugs.python.org/issue36771> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36771] Feature Request: An option to os.walk() to return os.DirEntry lists instead of just filenames/dirnames
CJ Kucera added the comment: I've started up a Github PR for this, btw, though IMO it's not really in a mergeable state yet: 1) I wasn't sure what to do about os.fwalk(), since that *doesn't* already generate DirEntry objects, and this change would introduce a small inconsistency between the two, functionalitywise. 2) Also wasn't sure what to do about unit tests, though I'll ponder that some more once I've got some time later. 3) The actual implementation is pretty trivial, but could perhaps be handled differently. The systems's still processing my CLA signature, as well, so there's that too. :) -- ___ Python tracker <https://bugs.python.org/issue36771> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36771] Feature Request: An option to os.walk() to return os.DirEntry lists instead of just filenames/dirnames
New submission from CJ Kucera : It'd be nice to have an option to os.walk which would return DirEntry objects in its return tuple, as opposed to just the string filenames/dirnames. (Or failing that, an alternate function which does so.) The function already uses os.scandir() internally, so the DirEntry objects already exist -- I assume it'd be a pretty easy change. At the moment, if I want to be efficient and use os.scandir() myself, I've got to basically reimplement os.walk(), which seems silly since os.walk is already calling scandir itself. -- components: Library (Lib) messages: 341210 nosy: apocalyptech priority: normal severity: normal status: open title: Feature Request: An option to os.walk() to return os.DirEntry lists instead of just filenames/dirnames type: enhancement versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue36771> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14044] IncompleteRead error with urllib2 or urllib.request -- fine with urllib, wget, or curl
CJ Kucera added the comment: Ah, well, actually I suppose I'll rescind that a bit - other pages about this bug around the internet had been claiming that the 'requests' module uses urllib in the backend and was subject to this bug as well, but after experimenting myself, it seems like if that IS the case, they're working around it somehow, because using requests makes this succeed 100% of the time. I probably should've tried that first! So anyway, there's a reasonable workaround, at least. Sorry for the bugspam! -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue14044> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14044] IncompleteRead error with urllib2 or urllib.request -- fine with urllib, wget, or curl
CJ Kucera added the comment:
I've just encountered this problem on Python 3.6, on a different URL. The
difference being that it's not encountered with EVERY page load, though I'd say
it happens with at least half:
import urllib.request
html = urllib.request.urlopen('http://www.basicinstructions.net/').read()
print('Succeeded!')
I realize that the root problem here may be an HTTP server doing something
improper, but I've got no way of fixing someone else's webserver. It'd be
really nice if there was a reasonable way of handling this in Python itself.
As mentioned in the original report, other methods of retreiving this URL work
without fail (curl/wget/etc). As it is, the only way for me to be sure of
retreiving the entire page contents is by looping until I don't get an
IncompleteRead, which is hardly ideal.
--
nosy: +apocalyptech
versions: +Python 3.6
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue14044>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: [python-list] Re: Strange crash issue on Windows w/ PyGTK, Cairo...
CJ Kucera wrote: Okay, I've got a reproducible testcase of this available up here: http://apocalyptech.com/pygtk-zlib/ I'm no longer *totally* convinced that it's a zlib issue... zlib's call actually returns a valid string, and the error happens later in the app. Hello, again, list. One last update on this, in case anyone happened to be curious about it. After talking with the PyCairo folks, it looks like this *was* a PyCairo issue after all, and the changes to the zlib call which had apparently fixed the issue were just incidental, basically, because of how the memory allocation ended up changing because of the modified vars. Anyway, a recent fix in PyCairo CVS seems to take care of it. zlib, as probably should have been expected, is vindicated! -CJ -- WOW: Flemmy| Happiness isn't good enough for me! I p...@apocalyptech.com | demand euphoria! 24.24.2.3171 | - Calvin -- http://mail.python.org/mailman/listinfo/python-list
Re: [python-list] Re: Strange crash issue on Windows w/ PyGTK, Cairo...
bieff...@gmail.com wrote: If you have worked with C/C++, you know that memory-related bugs can be very tricky. More than once - working with C code - I had crashes that disappeared if I just added a 'printf', because the memory allocation scheme changed and the memory corrupted was not anymore relevant. Well, you turned out to be dead right about this, as I suppose should have been pretty obvious given the nature of the problems I was having. Anyway, the issue turned out to be zlib.decompress() - for larger sets of data, if I wasn't specifying bufsize, the malloc()s that it was doing behind-the-scenes must have been clobbering memory. As soon as I specified bufsize, everything was totally kosher. Once I'm a bit more awake tomorrow I'll put together a testcase and send it in to the bug tracker. This does bring up one question: for larger chunks of data, is it More Appropriate to use a zlib decompression object instead of just passing it all through zlib.decompress()? Thanks, everyone... -CJ -- http://mail.python.org/mailman/listinfo/python-list
Re: Strange crash issue on Windows w/ PyGTK, Cairo...
CJ Kucera wrote: Anyway, the issue turned out to be zlib.decompress() - for larger sets of data, if I wasn't specifying bufsize, the malloc()s that it was doing behind-the-scenes must have been clobbering memory. As soon as I specified bufsize, everything was totally kosher. Okay, I've got a reproducible testcase of this available up here: http://apocalyptech.com/pygtk-zlib/ I'm no longer *totally* convinced that it's a zlib issue... zlib's call actually returns a valid string, and the error happens later in the app. I've yet to be able to engineer a crash using anything other than that cairo.ImageSurface.create_from_png() function, so it's possible that specifying bufsize in zlib.decompress() merely allocates memory in such a way that a bug in PyCairo doesn't come to light in that case. So, I'm not really sure if I should submit this to Python or PyGTK's tracker yet. Could someone check it out and let me know what you think? That'd be great. Thanks! As I mention on that page, removing import os and import sys will fix the issue on XP, though you can remove them on win2k and still see the crash. Thanks, CJ -- WOW: Flemmy| The ships hung in the sky in much the same p...@apocalyptech.com |way that bricks don't. - Douglas Adams, 24.24.2.3171 | _The Hitchhiker's Guide To The Galaxy_ -- http://mail.python.org/mailman/listinfo/python-list
Re: [python-list] Re: Strange crash issue on Windows w/ PyGTK, Cairo...
CJ Kucera wrote: Okay, I've got a reproducible testcase of this available up here: http://apocalyptech.com/pygtk-zlib/ Hello, two brief notes here: 1) Someone on the PyGTK list mentioned that I should really be using StringIO instead of my own hacky attempt at one, in there, and of course he was right. Replacing my class with StringIO doesn't result in any changed behavior, at least, so I can definitely rule out any weirdness that my own file-like class could have introduced. I've uploaded a new version of run.py to the above URL which includes that change. 2) It looks like specifying bufsize isn't actually the totally magic bullet for this. I had put in that fix to my application, so it was using bufsize, and things seemed to be working all right, but after four or five data loads, I got that Windows crash again, regardless. The fix does make the application stable enough that I'm not that worried about it, at least, but it doesn't seem to have totally bypassed whatever bug I'm running into. -CJ -- WOW: Flemmy| Happiness isn't good enough for me! I p...@apocalyptech.com | demand euphoria! 24.24.2.3171 | - Calvin -- http://mail.python.org/mailman/listinfo/python-list
Re: [python-list] Re: Strange crash issue on Windows w/ PyGTK, Cairo...
bieff...@gmail.com wrote: It looks like some of the C extension you are using is causing a segfault or similar in python interpreter (or it could be a bug in the interpreter itself, but it is a lot less likely). Okay... I assume by C extension you'd include the PyGTK stuff, right? (ie: pycairo, pygobject, and pygtk) Those are the only extras I've got installed, otherwise it's just a base Python install. Would a bad extension really cause this kind of behavior though? Specifically the working-the-first-time and crash-subsqeuent-times? Do C extensions contribute to the bytecode generated while compiling? I would suggest to fill the startup portion of your code with trace statements to try to understand which module function is the troublesome one, then go looking in the big tracking system of the module, try the newest version and ask on the dedicated mailing list if any. Are you talking about just throwing in various print statements, to find out where exactly it's dying? Or I see that there is an actual trace module in Python... I did do the former awhile ago and didn't find anything conclusive really. It was when approaching it from that angle that I stumbled across the case that if I simply renamed one of my functions, everything started working again. I'll do this a bit more once I've gotten the program down to a more manageable level. Making a small script that cabn reproduce the bug is also a very good idea, and will help speed-up the problem solution. Right, that's the goal. Right now it's still pretty unwieldy, still. I'll keep on it. Thanks for the response! -CJ -- WOW: Flemmy| Happiness isn't good enough for me! I p...@apocalyptech.com | demand euphoria! 24.24.2.3171 | - Calvin -- http://mail.python.org/mailman/listinfo/python-list
Strange crash issue on Windows w/ PyGTK, Cairo...
Hello list! I'm having a strange issue, and I'm not entirely certain yet where the actual problem is (ie, Python, PyGTK, or gtk+), but I figure I'll start here. Bear with me, this'll probably be a long explanation... I've been building an app which is meant to be run on both Linux and Windows. It uses PyGTK for its GUI, and the main area of the app is a gtk.DrawingArea which I draw on using PyCairo. I've been developing on Linux, and it works great on that platform, with no issues that I'm aware of. When running on Windows, though, the app exhibits the following behavior: 1) When the .py of the main file which runs the application GUI first gets compiled to a .pyc (ie: the first time it's run, or the first time after .py modification), the application runs totally fine, with no apparent problems. 2) Any attempt AFTER that, the application will start up, *start* to do its data-loading, but then almost immediately crash with an enigmatic python.exe has generated errors and will be closed by Windows. When it does so, there is no output whatsoever to the console that the application was launched from, and the crash doesn't always happen in exactly the same place. The pattern remains the same, though - if the .pyc needs to be compiled, the application works fine, but if not: boom. I've been steadily stripping the program down to what I hoped would be a small, reproducible app that I could post here, and I do intend to do so still, but it's rather slow going. For now, I was hoping to see if anyone's ever heard of behavior like this before, and might know what to do about it, or at least a possible avenue of attack. As I've been reducing the program down, I've encountered even stranger (IMO) behavior... In one instance, changing a function name seemed to make the program work. I took out the handler which draws my app's About box, and suddenly my problem went away. Occasionally I would remove a function and the app would suddenly *always* fail with that Windows crash error, and I'd have to put the function back in. Keep in mind, these are functions which *aren't being called anywhere.* Sometimes I could replace a function's entire contents with just pass and the app would suddenly behave properly, or not behave at all. It's almost as if whatever's doing the byte-compilation is getting screwed up somehow, and really small changes to parts of the file which aren't even being touched are having a huge impact on the application as a whole. It's seriously vexing, and certainly the oddest problems I've seen in Python. Windows versions I can reproduce this on: XP and win2k Python versions I've reproduced this on: Python 2.5.4 with: PyGTK 2.12.1-2-win32-py2.5 PyGObject 2.14.1-1.win32-py2.5 PyCairo 1.4.12-1.win32-py2.5 Python 2.6.1 with: PyGTK 2.12.1-3-win32-py2.6 PyGObject 2.14.2-2.win32-py2.6 PyCairo 1.4.12-2.win32-py2.6 gtk+ 2.12.9-win32-2 (from http://sf.net/projects/gladewin32 , which is the version linked to from pygtk.org) The 2.6 Python stuff I've actually only tried on win2k so far, not XP, though given my history with this, I suspect that that wouldn't make a difference. Since gtk+ is the one bit of software that hasn't been swapped out for another version, I suppose that perhaps that's where the issue is, but it seems like Python should be able to at least throw an Exception or something instead of just having a Windows crash. And having it work the FIRST time, when the .pyc's getting compiled, is rather suspicious. Anyway, I'll continue trying to pare this app down to one manageable script which I can post here, but until then I'd be happy to hear ideas from anyone else about this. Thanks! -CJ -- http://mail.python.org/mailman/listinfo/python-list
