[issue31170] expat: utf8_toUtf8 cannot properly handle exhausting buffer
Lin Tian added the comment: Reactive this issue as to let you know that libexpat has confirmed and fixed the bug and they are interested in porting the fix to python. Reactive this in case you want to know what's going on and make a decision accordingly. (Sorry, I'm not very familiar with process here) -- status: closed -> open ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue31170> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31170] expat: utf8_toUtf8 cannot properly handle exhausting buffer
New submission from Lin Tian:
utf8_toUtf8(const ENCODING *UNUSED_P(enc),
const char **fromP, const char *fromLim,
char **toP, const char *toLim)
{
char *to;
const char *from;
const char *fromLimInitial = fromLim;
/* Avoid copying partial characters. */
align_limit_to_full_utf8_characters(*fromP, );
for (to = *toP, from = *fromP; (from < fromLim) && (to < toLim); from++, to++)
*to = *from;
*fromP = from;
*toP = to;
if (fromLim < fromLimInitial)
return XML_CONVERT_INPUT_INCOMPLETE;
else if ((to == toLim) && (from < fromLim))
// <= Bug is here. In case (to == toLim), it's possible that
//from is still pointing to partial character. For example,
//a character with 3 bytes (A, B, C) and form is pointing to C.
//It means only A and B is copied to output buffer. Next
//scanning will start with C which could be considered as invalid
//byte and got dropped. After this, only "AB" is kept in memory
//and thus it will lead to invalid continuation byte.
return XML_CONVERT_OUTPUT_EXHAUSTED;
else
return XML_CONVERT_COMPLETED;
}
--
components: Library (Lib)
messages: 300043
nosy: Lin Tian
priority: normal
severity: normal
status: open
title: expat: utf8_toUtf8 cannot properly handle exhausting buffer
type: behavior
versions: Python 3.6, Python 3.7
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue31170>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Convert from unsigned long long to PyLong
HI There,
I'm using MySQLdb as the MySQL client. Recently I got a weird problem of this
library. After looking into it, I suspect the problem may related to the
conversion from unsigned long to PyLongObject.
Here is the detail, If you are familiar with MySQLdb, the following snippet is
a way to query the data from MySQL:
connection = MySQLdb.connect(...)
connection.autocommit(True)
try:
cursor = connection.cursor()
if not cursor.execute(sql, values) > 0:
return None
row = cursor.fetchone()
finally:
connection.close()
return row[0]
Sometimes the return value of execute method would be 18446744073709552000 even
there is no matched data available. I checked the source code of the library,
the underlying implementation is
https://github.com/farcepest/MySQLdb1/blob/master/_mysql.c#L835,
static PyObject *
_mysql_ConnectionObject_affected_rows(
_mysql_ConnectionObject *self,
PyObject *args)
{
if (!PyArg_ParseTuple(args, "")) return NULL;
check_connection(self);
return
PyLong_FromUnsignedLongLong(mysql_affected_rows(&(self->connection)));
}
And here is the official doc for mysql_affected_rows
http://dev.mysql.com/doc/refman/5.7/en/mysql-affected-rows.html.
Let me give a superficial understanding, please correct me if I were wrong.
In a 64-bit system, the mysql_affected_rows is supposed to return a number of
unsigned long, which means the range should be 0 ~ 2^64 (18446744073709551616),
How could it be possible the function PyLong_FromUnsignedLongLong return a
converted value larger than 2^64, that's what I don't understand.
Does anyone have some ideas of it?
The versions of the components I used:
Python: 2.7.6
MySQL 5.7.11
MySQLdb 1.2.5
Thanks
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to handle exceptions properly in a pythonic way?
在 2015年11月4日星期三 UTC-6下午10:18:33，zlju...@gmail.com写道： > > Which would you prefer? > > So if I am just checking for the ConnectionError in get_html and a new > exception arises, I will have traceback to the get_html function showing that > unhandled exception has happened. > Now I have to put additional exception block for managing the new exception > in the get_html function and I am covered. > > Is that what you wanted to say? Hi, If I may, I feel you are trying to address a few separate questions, although they do relate to each other. 1. Coding Design: with the try/except block inside or outside the function 2. Exception Handling: What to do with new un-anticipated exceptions 3. How to record the exception for reference: logging or special value My feeling is: 1. Personally prefer to put try/except block outside the function, to keep the code clean and easy to follow. 2. I would suggest follow the Test Driven Development (TDD) approach. You are not able to anticipate all types of possible exceptions. However, I'm sure you have a pretty good idea about what exceptions are more likely to happen and cause you problem. In that case, just develop your code to pass these tests, and refactor it in the future if really needed. Not necessary to push your code to be perfect, able to handle every possible exception. 3. Kind of personal choice here, since no matter which way you go, you do need to follow it up and deal with them. Probably I would log it just for record keeping purpose. As long as you follow up and trace back to the root cause, they seem to serve the same goal. Hope this helps... All the best, Tian -- https://mail.python.org/mailman/listinfo/python-list
Re: How to handle exceptions properly in a pythonic way?
Hi, If I may, I feel you are tying to address a few questions at the same time, although they are related 1. Where to put the try/except block, inside or outside the function 2. How to deal with un-anticipated exceptions 3. How to keep record My personal feelings are: 1. Kind of prefer try/except block outside the function though. This way it looks clean and easy to follow. In terms of logging the url, since you pass it to the function anyway, there should be ways to keep this option, even with the block written outside the function. 2. From code development perspective, would suggest you follow Test Driven Development (TDD) approach. Nobody can anticipate all of the possible outcomes, but I'm sure you have a pretty good idea about the most likely scenarios, and just make sure that your code is suitable for these scenarios. When future new exceptions arise, you can always refactor your code as needed. 3. Feel it's a personal choice here as being pointed out earlier. No mater which way you go, you just have to follow it up and deal with it, trace it back to the root cause. I kind of prefer logging, so to keep a good record for myself, and you can make it very informative. Hope this helps. All the best, Tian -- https://mail.python.org/mailman/listinfo/python-list
Re: raw_input and break
在 2015年11月4日星期三 UTC-6下午3:45:09，input/ld...@casema.nl写道：
> I have an continues loop with "while True:"
> Now I want to use "raw_input" and when I press "s" on the keybord that it
> will
> "break" the continues loop.
>
> I tried:
> choices = raw_input
> if choises == s:
> break
>
> But even when I not press "s" it "break"
> I want that I not press "s" the script continues.
>
> Any ideas ?
>
> Thanks
>
>
>
> --
> - --- -- -
> Posted with NewsLeecher v7.0 Beta 2
> Web @ http://www.newsleecher.com/?usenet
> --- - -- -
while True:
inp = input("Enter whatever you want, letter 's' to stop: ")
if inp == 's':
print("Program stopped.")
break
print("You just entered:", inp)
Is this along the line that you are thinking? As pointed out earlier, it will
be much easier for people to help if you can share the exact code you put into
and the error messages as well. Thanks and good luck!
All the best,
Tian
--
https://mail.python.org/mailman/listinfo/python-list
question on string object handling in Python 2.7.8
Hi, There are 2 statements: A: a = ‘h’ B: b = ‘hh’ According to me understanding, A should be faster as characters would shortcut this 1-byte string ‘h’ without malloc; B should be slower than A as characters does not work for 2-byte string ‘hh’, which triggers the malloc. However, when I put A/B into a big loop and try to measure the performance using cProfile, B seems always faster than A. Testing code: for i in range(0, 1): a = ‘h’ #or b = ‘hh’ Testing cmd: python -m cProfile test.py So what is wrong here? B has one more malloc than A but is faster than B? Thanks, Dave -- https://mail.python.org/mailman/listinfo/python-list
[issue22870] urlopen timeout failed with SSL socket
Dave Tian added the comment: Alright. The issued URL of my case is here: www.5giay.vn Nor am I a ssl dev...Here is what happens after further debugging: PySSL_SSLread() returns 1/2 bytes without any error/timeout per call. readline() in socket.py keeps looping. Occasionally, it may break out because of a newline recv’d. However, readheaders() in httplib.py repeats the readline() again because of another while True loop. I have no idea why I’ve got only 1/2 bytes recv’d from SSLread(). I am using my Mac OS for testing. Hope this is clear for further investigation if anyone would like to dig in. Dave Tian dave.jing.t...@gmail.com On Nov 16, 2014, at 2:38 AM, R. David Murray rep...@bugs.python.org wrote: R. David Murray added the comment: I won't be the one, as I'm not conversant with the ssl C code. What would be helpful right now would be a recipe for reproducing the problem. -- nosy: +alex, pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22870 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22870 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22870] urlopen timeout failed with SSL socket
Dave Tian added the comment: Hi David, Thanks for your quick response. I have tried Python 3.4.2 using urllib.request.urlopen() - still not working. Below is the backtrace. I am not sure if this is a bug of PySSL_SSLread, which returns nothing yet without timeout. If you want me to dig into this deeper, just let me know. Thanks, Dave Tian dave.jing.t...@gmail.com obj=urllib.request.urlopen(url, timeout=20) ^CTraceback (most recent call last): File stdin, line 1, in module File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 153, in urlopen return opener.open(url, data, timeout) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 461, in open response = meth(req, response) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 571, in http_response 'http', request, response, code, msg, hdrs) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 493, in error result = self._call_chain(*args) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 433, in _call_chain result = func(*args) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 676, in http_error_302 return self.parent.open(new, timeout=req.timeout) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 455, in open response = self._open(req, data) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 473, in _open '_open', req) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 433, in _call_chain result = func(*args) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 1217, in https_open context=self._context, check_hostname=self._check_hostname) File /Users/daveti/Python-3.4.2/Lib/urllib/request.py, line 1177, in do_open r = h.getresponse() File /Users/daveti/Python-3.4.2/Lib/http/client.py, line 1172, in getresponse response.begin() File /Users/daveti/Python-3.4.2/Lib/http/client.py, line 375, in begin self.headers = self.msg = parse_headers(self.fp) File /Users/daveti/Python-3.4.2/Lib/http/client.py, line 261, in parse_headers line = fp.readline(_MAXLINE + 1) File /Users/daveti/Python-3.4.2/Lib/socket.py, line 371, in readinto return self._sock.recv_into(b) File /Users/daveti/Python-3.4.2/Lib/ssl.py, line 746, in recv_into return self.read(nbytes, buffer) File /Users/daveti/Python-3.4.2/Lib/ssl.py, line 618, in read v = self._sslobj.read(len, buffer) KeyboardInterrupt On Nov 15, 2014, at 3:36 AM, R. David Murray rep...@bugs.python.org wrote: R. David Murray added the comment: It sounds like the bug is that PySSL_SSLread didn't raise the timeout? Any idea if this is still a problem in python3? (Could possibly have changed on trunk as well, as SSL is being updated in 2.7.9.) -- nosy: +r.david.murray ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22870 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22870 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22870] urlopen timeout failed with SSL socket
New submission from Dave Tian: Hi there, Recent urlopen with timeout did not work. Below is the back trace. After digging into the Python lib, the root cause is found - within the socket.py, self._sock.recv(), under a 'while True' loop, tried to retrieve sth from the under-layer SSL socket. However, the under-layer PySSL_SSLread() neither returned anything useful nor reported an exception, such as timeout. Because of this 'while True' loop, urlopen got stuck. My temp fix is to add timeout within this issued loop of the socket.py. However, there are other similar loops ranging from urllib2, httplib to socket...and I do not think anything wrong with these loops considering the system socket programming practice. Instead, I am thinking if we could add a new timeout value into the socket object. This new timeout starts with the same value as the user-defined timeout but decrease for the same socket operation. For the case I have encountered: while True: some_sock_recv() # decrease the timeout value within the function call This should guarantee the timeout timely even within the 'while True' loop. Thanks, Dave http://davejingtian.org obj=urllib2.urlopen(url, timeout=20) ^CTraceback (most recent call last): File stdin, line 1, in module File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 127, in urlopen return _opener.open(url, data, timeout) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 410, in open response = meth(req, response) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 523, in http_response 'http', request, response, code, msg, hdrs) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 442, in error result = self._call_chain(*args) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 382, in _call_chain result = func(*args) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 629, in http_error_302 return self.parent.open(new, timeout=req.timeout) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 404, in open response = self._open(req, data) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 422, in _open '_open', req) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 382, in _call_chain result = func(*args) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 1222, in https_open return self.do_open(httplib.HTTPSConnection, req) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py, line 1187, in do_open r = h.getresponse(buffering=True) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py, line 1045, in getresponse response.begin() File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py, line 441, in begin self.msg = HTTPMessage(self.fp, 0) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/mimetools.py, line 25, in __init__ rfc822.Message.__init__(self, fp, seekable) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/rfc822.py, line 108, in __init__ self.readheaders() File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py, line 280, in readheaders line = self.fp.readline(_MAXLINE + 1) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py, line 476, in readline data = self._sock.recv(self._rbufsize) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py, line 241, in recv return self.read(buflen) File /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py, line 160, in read return self._sslobj.read(len) KeyboardInterrupt -- components: Library (Lib) messages: 231160 nosy: daveti priority: normal severity: normal status: open title: urlopen timeout failed with SSL socket type: behavior versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue22870 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: How close program in the wxPython?
I know how to capture window close event, but what i want to do is to close window using commands. now i am simply using sys.exit() to close the whole program. but it seems that window close events will are be generated in this case. is there any function in wxpython that i can use to close application window, yet generate the windows close event? -- http://mail.python.org/mailman/listinfo/python-list
How close program in the wxPython?
I have made a program in wxpython, but how could i exit the program? I am using wxFrame for my window, what is the function to close the program? Thanks!! -- http://mail.python.org/mailman/listinfo/python-list
Re: How to execute a cmd line program without invoking console window?
this is very useful, thanks very much! -- http://mail.python.org/mailman/listinfo/python-list
Problem in designing a global directory in python
I want to create a object directory called Context in my program, which is based on a dict to save and retrieve values/objects by string-type name. I have the definition like this: utils.py global sysctx class Context: def __init__(self): def set(self, name, obj, overwrite=True): def get(self, name): def has(self, name): def init(): global sysctx sysctx = Context() def getContext(): global sysctx return sysctx - init() is called somewhere at the beginning of the program. In other modules, i want to use this in the following manner, from utils import * getContext().set(...) but SOMETIMES I met following error located in getContext() NameError: global name 'sysctx' is not defined I found that when a module is in the same directory as utils.py, when I can simply use utils for importing, there is no such problem. But when i was writing a module in a deeper directory than utils.py, which has to use the full module name for importing, such as: from myproj.utils import * getContext().set(...) I got this error. What should I do to correct that? -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem in designing a global directory in python
I have tried using sysctx=None instead of global sysctx, but it doesn't work either. It seems my initialization work in the previous calling of init() has no persistent effect when utils is imported using from myproj.utils import getContext. What's weird, when a module is in the same directory as utils.py, where I can simply use utils for importing, there is no such problem. Any other suggestions? -- http://mail.python.org/mailman/listinfo/python-list
How to execute a cmd line program without invoking console window?
In Windows, I have been simply using os.system() to run command line program in python. but there will be a black console window. How can I run the program without invoking that window? i guess there are some function with which I can redirect the output? -- http://mail.python.org/mailman/listinfo/python-list
How to organize source code and imports???
I am writing a python program which needs to support some plug-ins. I have an XML file storing some dynamic structures. XML file records some class names whose instance needs to be created in the run time while parsing the XML file. I wonder what is the best solution for this problem? I also have some problem about the import. How should I design my packages? Say, I have all code locates at c:\projects\sami, c:\project is in my PYTHONPATH environment variable. Suppose my folder structure is like this: c: projects\ -this directory is in PYTHONPATH sami\ __init__.py main.py xmlparser.py window.py proc.py support\ __init__.py helper.py plugins\ __init__.py BaseClass.py---no instance for this one ExtClassA.py ExtClassB.py ExtClassC.py ExtClassD.py Each file in \projects\sami\plugins contains a class with a same name as the file, (ExtClassA.py has class ExtClassA), the instance of these classes need to be created at runtime while functions in xmlparser.py is parsing an XML file. main.py is the start point of the program. Other files in the \projects\sami\ and projects\sami\support are supporting modules. 1. What should I write in each __init__.py ??? In main.py, if I need functions in proc.py, should I write import proc? If I need functions in helper.py, can i write import support.helper?? what else should I do to support all these? 2. What is the best way to make instance of a class from a string type name? One method I have successfully tried is using from SomeWhere import *, then get class from globals() and make instance. But How can I import all ExtClass?.py? Where should I write these import codes? -- http://mail.python.org/mailman/listinfo/python-list
newbie question: how to get the class instance given a module object?
I have a module called ModuleA.py, in which there is a class called Dog, what should I put in the part to get the instance of class Dog??? import ModuleA classname = Dog module = globals()[ModuleA] classobj = ??? ---using classname instanct = classobj() -- http://mail.python.org/mailman/listinfo/python-list
Re: How to create an object instance from a string??
This is very useful, thanks! -- http://mail.python.org/mailman/listinfo/python-list
How two modules call functions defined in each other?
I am python beginner, I have a question about the interdependence of modules. For example, when I have two modules: module1.py - def plus(x): return add(x,1) module2.py - def add(x,y): return x+y def plus2(x): return plus(x)+1 How should I write import in both files? What about the global varibals? is there anything like extern keyword in C? or python has some other solutions? -- http://mail.python.org/mailman/listinfo/python-list
How to write python plug-ins for your own python program?
I am writing an audio game using Python. in this game you can apply some sound effects for the clips you have recorded. I want to make this function extensible. I want user to be able to add new sound effect plug-ins in the future. I want the plug-in to be a simple python code (text file) and a description file. I will set some rules for plug-in writing (like you must inherit some class and implement some method). I hope plugin can be added while original program is running. Is there any good method to read in python code and test availability and invoke the functions inside? Thanks Tian -- http://mail.python.org/mailman/listinfo/python-list
Re: pyFMOD problem
I think I did have put the DLL file in the correct place, but I still cannot make it work. same problem. I checked my version of pyfmod, ctypes, they are all the newest one. can you tell me the version you are using? Thanks -- http://mail.python.org/mailman/listinfo/python-list
pyFMOD problem
I am trying to use pyfmod in python to manipulate sound. i have installed pyfmod, ctype, numarray (if they are necessary), i also copied fmod.dll to python/DLLs as well as windows/system32, but when i tried import pyFMOD I got: Traceback (most recent call last): File pyshell#0, line 1, in -toplevel- import pyFMOD File C:\Python24\Lib\site-packages\pyFMOD.py, line 177, in -toplevel- _FSOUND_Sample_Load = getattr(fmod, [EMAIL PROTECTED]) File C:\Python24\Lib\site-packages\ctypes\__init__.py, line 323, in __getattr__ func = self._StdcallFuncPtr(name, self) AttributeError: function '[EMAIL PROTECTED]' not found - How can I deal with this? Thanks! -- http://mail.python.org/mailman/listinfo/python-list
