[issue47221] Bug or bad performance
Cezary Wagner added the comment: Some more experiments: import dis import timeit REPEATS = 100 def test1(): selected = [] for i in range(REPEATS): if i >= 25 and i <= 75: selected.append(i) return selected def test2(): selected = [] for i in range(REPEATS): if 25 <= i <= 75: selected.append(i) return selected def test3(): return [x for x in range(REPEATS) if x >= 25 and x <= 75] def test4(): return [x for x in range(REPEATS) if 25 <= x <= 75] def test(f): print(dis.dis(f.__code__.co_code)) print(timeit.timeit(f)) test(test1) test(test2) test(test3) test(test4) Result: 0 BUILD_LIST 0 2 STORE_FAST 0 (0) 4 LOAD_GLOBAL 0 (0) 6 LOAD_GLOBAL 1 (1) 8 CALL_FUNCTION1 10 GET_ITER >> 12 FOR_ITER15 (to 44) 14 STORE_FAST 1 (1) 16 LOAD_FAST1 (1) 18 LOAD_CONST 1 (1) 20 COMPARE_OP 5 (>=) 22 POP_JUMP_IF_FALSE 21 (to 42) 24 LOAD_FAST1 (1) 26 LOAD_CONST 2 (2) 28 COMPARE_OP 1 (<=) 30 POP_JUMP_IF_FALSE 21 (to 42) 32 LOAD_FAST0 (0) 34 LOAD_METHOD 2 (2) 36 LOAD_FAST1 (1) 38 CALL_METHOD 1 40 POP_TOP >> 42 JUMP_ABSOLUTE6 (to 12) >> 44 LOAD_FAST0 (0) 46 RETURN_VALUE None 4.56567799025 0 BUILD_LIST 0 2 STORE_FAST 0 (0) 4 LOAD_GLOBAL 0 (0) 6 LOAD_GLOBAL 1 (1) 8 CALL_FUNCTION1 10 GET_ITER >> 12 FOR_ITER19 (to 52) 14 STORE_FAST 1 (1) 16 LOAD_CONST 1 (1) 18 LOAD_FAST1 (1) 20 DUP_TOP 22 ROT_THREE 24 COMPARE_OP 1 (<=) 26 POP_JUMP_IF_FALSE 18 (to 36) 28 LOAD_CONST 2 (2) 30 COMPARE_OP 1 (<=) 32 POP_JUMP_IF_FALSE 25 (to 50) 34 JUMP_FORWARD 2 (to 40) >> 36 POP_TOP 38 JUMP_ABSOLUTE6 (to 12) >> 40 LOAD_FAST0 (0) 42 LOAD_METHOD 2 (2) 44 LOAD_FAST1 (1) 46 CALL_METHOD 1 48 POP_TOP >> 50 JUMP_ABSOLUTE6 (to 12) >> 52 LOAD_FAST0 (0) 54 RETURN_VALUE None 5.639823407506 0 LOAD_CONST 1 (1) 2 LOAD_CONST 2 (2) 4 MAKE_FUNCTION0 6 LOAD_GLOBAL 0 (0) 8 LOAD_GLOBAL 1 (1) 10 CALL_FUNCTION1 12 GET_ITER 14 CALL_FUNCTION1 16 RETURN_VALUE None 3.879290759425 0 LOAD_CONST 1 (1) 2 LOAD_CONST 2 (2) 4 MAKE_FUNCTION0 6 LOAD_GLOBAL 0 (0) 8 LOAD_GLOBAL 1 (1) 10 CALL_FUNCTION1 12 GET_ITER 14 CALL_FUNCTION1 16 RETURN_VALUE None 3.859126637943 -- ___ Python tracker <https://bugs.python.org/issue47221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47221] Bug or bad performance
New submission from Cezary Wagner : I am experienced programmer 10y+ - that is very very strange performance problem when I play Python timeit with my son :) three way operator a <= x <= b is slower than a <= x and x <= b. It looks like wrong implementation since it is impossible that two separate check is faster that one check (with two low level check in C). import timeit REPEATS = 100 def test1(): selected = [] for i in range(REPEATS): if i >= 25 and i <= 75: selected.append(i) return selected def test2(): selected = [] for i in range(REPEATS): if 25 <= i <= 75: selected.append(i) return selected print(timeit.timeit(test1)) print(timeit.timeit(test2)) Result is on Windows 10. 4.42894768389 4.906247778 -- components: Interpreter Core messages: 416699 nosy: Cezary.Wagner priority: normal severity: normal status: open title: Bug or bad performance type: performance versions: Python 3.10 ___ Python tracker <https://bugs.python.org/issue47221> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment: As far as I know os.stat() resets d.stat() maybe should be added some option to d.stat() to force update(). d.stat(nt_force_update=True). I am not sure if os.path.getmtime() can reset d.stat(). os.stat() is 2x times slower than os.path.getmtime() and os.path.getmtime is 16x slower than d.stat(). MAJOR PROBLEM is PERFORMANCE of os.stat() since for directories with 1000 files it takes big number of seconds to read all stats - something wrong is here I think since Windows Explorer is doing it very fast. So I can not use os.stat() ONLY and it complicates code since I need to use os.stat() after d.stat() if files is OLDER THAN because if I use os.stat() the most program time will be these calls. Do you know which code makes such reset of d.stat()? If there is not possible optimization of there is need DOCUMENTATION update because it is really hard to understand why it is not working under windows some REMARKS can help me and others. I have still believe that some optimization is possible for Windows. Maybe it can be force to read stat by os.scandir(force_scan_stat=True) so all directory entries will be have cached stats before d.stat() is called. It can be faster I think since less calls from Python and probably better Windows API for it and same for Linux. I will study C code later if it is possible or write some snippet. -- ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment: I think we can assume that NTFS is priority since that is the most used option. I can not discuss what with FAT32 or FAT since I am not the best in this domain (in NTFS I am not the best too now). Whatever I think that system must do allocation for open files to avoid conflicts so it can be tracked but how? Possible solutions is some extra function, argument for Windows - which makes cache dirty between calls. It is very dirty proposal - I need to think if it is good. Even used names is ugly I need think more about it. My imagination tells me that it can be good direction. dir_entry.stat(nt_force_cache_refresh=True) - it can be good for specific entries. os.scandir(nt_force_cache_refresh=True) - it is sometimes not need for all entries I am thinking: dir_entry.stat(nt_force_cache_refresh=True) should be faster than os.stat(dir_entry.path) instead dir_entry.stat() which not works fo open files. os.scandir(nt_force_cache_refresh=True) should be faster than dir_entry.stat(nt_force_cache_refresh=True) and dir_entry.stat() will work for open files. It is simpler to understand that Windows is different if such extra attribute must be added at all. nt_force_cache_refresh can add to dir_entry some information that .stat() should not use cache. Then best will be to not use nt_force_cache_refresh for open files - maybe you will find the way to detect open files in external application. I think Windows API allow to check if file is open - as far as remember sysinternals tools can do this so there some API for it I think. See this tool: https://docs.microsoft.com/en-us/sysinternals/downloads/handle - maybe there is source code for it or you can learn for it. Maybe you can check if file is open with use this API before dir_entry.stat() I do want to force any solution but just share some rough ideas. -- ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment:
I read some comments os.flush() or os.fsync() can be unrelated to problem.
External application can be written in C# or whatever you want.
Under Windows (not Linux) - modification dates will be stalled in such sequence.
os.scandir()
dir_entry.stat() # let it be dir_entry.path == 'test.txt'
dir_entry.stat().st_mtime # will be for example 1
os.scandir()
dir_entry.stat() # let it be dir_entry.path == 'test.txt'
dir_entry.stat().st_mtime # will be STALLED for example 1
Under Windows (not Linux) - modification dates will be refreshed in such
sequence.
os.scandir()
dir_entry.stat() # let it be dir_entry.path == 'test.txt'
dir_entry.stat().st_mtime # will be for example 1
os.stat('test.txt') # this code do something and it is not stalled in next call
os.scandir()
dir_entry.stat() # let it be dir_entry.path == 'test.txt'
dir_entry.stat().st_mtime # will be CHANGED for example 2
--
___
Python tracker
<https://bugs.python.org/issue41106>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment: Use case - detection of changes in open files is very important - log scanning - synchronization ... I think that first of all it is need good unit test to detect this problem (rare edge case - probably it is missed because hard to imagine that it can not work when file is open - I will miss this I think). It should work like this. First program is writing file under Windows and second program (unit test) is running os.scandir() if repeated os.scandir() detect changes it is O.K. (same like in Linux). To make it simpler it can be unit test in one program. 1. Open test file in test directory. 2. os.scandir() in test directory. 3. Some writes to test file (f.write() with and without flush, ... - to be defined what is sufficient to test). 4. os.scandir() in test directory - if change detected it O.K. 5. f.close() I do not know Windows API now but I think we can detect id directory is changed between scans or we can detect if file is open (it is rare situation - rare edge case) in 90% all files will be closed. So if all files is closed current os.scandir() maybe is good (not I do not understand implementation to evaluate it correclty) and when one of file or more there is need another implementation which will detect modification. If you think I missed something please comment. You are welcome. -- ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment: I do some test on linux all works - changes are detected and os.scandir() works but in Windows not - probably there is not unit test which check if os.scandir() is working on open files for writing. f.flush() no matter since file can be changed in external Python/Java/C#/C++, ... application - anyone can write logs in Windows. I will explain it in next comment. I just write this code to show only problem. Result from linux STAT = False so only repeat os.scandir() calls. Modification are detected correctly in Linux but not in Windows. [wagnecaz@nsdptrms01 ~]$ python3 s03_dir_entry.py dir_entry.stat() /home/wagnecaz/test.txt 1593085189.1000397 since last change 0.00111671508789 2020-06-25 13:39:49.101368 1593085189.101368 dir_entry.stat() /home/wagnecaz/test.txt 1593085189.1000397 since last change 1.0028572082519531 2020-06-25 13:39:50.103111 1593085190.103111 dir_entry.stat() /home/wagnecaz/test.txt 1593085190.1020408 since last change 1.0026073455810547 2020-06-25 13:39:51.104881 1593085191.104881 dir_entry.stat() /home/wagnecaz/test.txt 1593085191.104042 since last change 1.0023958683013916 2020-06-25 13:39:52.106793 1593085192.106793 dir_entry.stat() /home/wagnecaz/test.txt 1593085192.106043 since last change 1.0023260116577148 2020-06-25 13:39:53.108582 1593085193.108582 dir_entry.stat() /home/wagnecaz/test.txt 1593085193.1080444 since last change 1.0021436214447021 2020-06-25 13:39:54.110500 1593085194.1105 dir_entry.stat() /home/wagnecaz/test.txt 1593085194.1100454 since last change 1.0013866424560547 2020-06-25 13:39:55.111684 1593085195.111684 dir_entry.stat() /home/wagnecaz/test.txt 1593085195.1110466 since last change 1.0022354125976562 2020-06-25 13:39:56.113542 1593085196.113542 dir_entry.stat() /home/wagnecaz/test.txt 1593085196.1130476 since last change 1.0021603107452393 2020-06-25 13:39:57.115450 1593085197.11545 dir_entry.stat() /home/wagnecaz/test.txt 1593085197.1140487 since last change 1.003014326095581 Change is done every 1s and detected in Linux in Windows it is stalled. 2020-06-25 13:39:58.117287 1593085198.117287 dir_entry.stat() /home/wagnecaz/test.txt 1593085198.11605 since last change 1.002938985824585 2020-06-25 13:39:59.119224 1593085199.119224 dir_entry.stat() /home/wagnecaz/test.txt 1593085199.118051 since last change 1.0027978420257568 2020-06-25 13:40:00.121166 1593085200.121166 -- ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment: One hint more. Start of new process os.scandir() give invalid modification date for file open for writing until external tool is not called (like explorer, touch, etc.). So (log open for writing and write is done between 1, 2): 1. Run program with os.scandir() -> dir_entry.stat().st_mtime() = t1. 2. Run program with os.scandir() -> dir_entry.stat().st_mtime() = t1. Modification is stalled. Another scenario (log open for writing and write is done between 1, 3): 1. Run program with os.scandir() -> dir_entry.stat().st_mtime() = t1. 2. touch -> dir_entry.path 3. Run program with os.scandir() -> dir_entry.stat().st_mtime() = t2. Modification works. -- ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
Cezary Wagner added the comment: Extra file for for tests with: DO_STAT = False See not changes but file was writing every second. If os.stat() run all between call os.scandir() all works. C:\root\Python38\python.exe C:/Users/Cezary.Wagner/PycharmProjects/dptr-monitoring-v2/sandbox/python/s13_dir_entry/s03_dir_entry.py dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 0.0009987354278564453 2020-06-24 18:57:52.911980 1593017872.91198 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 1.0078418254852295 2020-06-24 18:57:53.918823 1593017873.918823 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 2.0103507041931152 2020-06-24 18:57:54.921332 1593017874.921332 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 3.023340940475464 2020-06-24 18:57:55.934322 1593017875.934322 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 4.036783933639526 2020-06-24 18:57:56.947765 1593017876.947765 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 5.049667835235596 2020-06-24 18:57:57.960649 1593017877.960649 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 6.063947916030884 2020-06-24 18:57:58.974929 1593017878.974929 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 7.0797247886657715 2020-06-24 18:57:59.990706 1593017879.990706 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 8.091670751571655 2020-06-24 18:58:01.002652 1593017881.002652 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 9.1053147315979 2020-06-24 18:58:02.016296 1593017882.016296 dir_entry.stat() T:\\test.txt 1593017872.9109812 since last change 10.120086908340454 2020-06-24 18:58:03.031068 1593017883.031068 -- Added file: https://bugs.python.org/file49260/s03_dir_entry.py ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41106] os.scandir() Windows bug dir_entry.stat() not works on file during writing.
New submission from Cezary Wagner : I have problem with change detection of log during writing under Windows (normal fs and windows share). Probably bad order of Windows API calls - no idea. Test program is attached. You can reproduce it. Try with os.scandir() without os.stats() and os.stat(). Source code responsible for it is probably this -> I do not understand CPython code -> https://github.com/python/cpython/blob/master/Modules/posixmodule.c. Here is full description - many test was done. # os.scandir() Windows bug dir_entry.stat() not works on file during writing. # Such files is for example application log. # No problem with os.stat() # Call of os.stat() before os.scandir() -> dir_entry.stat() is workaround. # Open file during writing other program "fixes" dir_entry.stat(). # Get properties on open file during writing "fixes" dir_entry.stat(). # Notice that I run os.scandir() separately so dir_entry.stat() is not cached. # Steps to reproduce lack of modification update: # 1. Close all explorers or other application using PATH (it has impact). # 2. Set PATH to test folder can be directory or windows share. # 3. Run program without DO_STAT (False). # # Alternative steps (external app force valid modification date): # 4. run 'touch' or 'echo' on file should "fix" problem. 'echo' will throw error not matter. # # Alternative scenario (os.stat() force valid modification date - very slow): # 3. Run program without DO_STAT (True). No problems. # # Error result: # Modification date from dir_entry.stat() is stalled (not changing after modification) # if os.stat() or other Windows application not read file. # # Excepted result: # Modification date from dir_entry.stat() is update from separate calls os.scandir() # or cached if it is same os.scandir() call. # # Notice that os.scandir() must be call before dir_entry.stat() to avoid caching as described in documentation. # And this is done but not work on files during writing.. # # Ask question if you have since is very hard to find bug. -- components: Windows files: s03_dir_entry.py messages: 372264 nosy: Cezary.Wagner, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: os.scandir() Windows bug dir_entry.stat() not works on file during writing. type: crash versions: Python 3.8 Added file: https://bugs.python.org/file49259/s03_dir_entry.py ___ Python tracker <https://bugs.python.org/issue41106> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment: C:\root\Python\Python38\python.exe "C:/Users/Cezary Wagner/PycharmProjects/chess-lichess-eval-parse/sandbox/s03_create_tree/s03_python_bug.py" seek 18446744073709554618 3003 seek 0 75 114 145 165 181 205 227 246 265 290 315 328 365 387 411 413 18446744073709554618 next 3003 seek 18446744073709554618 3003 Process finished with exit code 0 -- ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment:
I tested is and it is not "state cookie" but it is "absolute position" and it
is "stateless".
> The I/O stack in Python 3 does not use C FILE streams, and this issue is not
> related to Windows. TextIOWrapper.tell returns a "cookie" based on the
> decoder state:
I will study:
https://github.com/python/cpython/blob/3.8/Modules/_io/textio.c#L2589 - Thank
you for reference.
See this test code (wrong seek is good - decoder has not state before - first
line of program).
You can swap this two fragments and it still works.
print('seek 18446744073709554618')
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
pgn.seek(18446744073709554618)
while pgn.tell() != 3003:
pgn.readline()
print(pgn.tell())
print()
print('seek 0')
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
pgn.seek(0)
while pgn.tell() != 18446744073709554618:
pgn.readline()
print(pgn.tell())
pgn.readline()
print('next', pgn.tell())
print('seek 18446744073709554618')
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
pgn.seek(18446744073709554618)
while pgn.tell() != 3003:
pgn.readline()
print(pgn.tell()))
--
___
Python tracker
<https://bugs.python.org/issue39962>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment: > The I/O stack in Python 3 does not use C FILE streams, and this issue is not > related to Windows. TextIOWrapper.tell returns a "cookie" based on the > decoder state: That can big problem when I use serialization if f.tell() is "cookie". When I serialize it and run program again f.seek() will not works. I will test it but I think that can be big problem since this behavior is very unclear and non standard (comparing to C++/C#/Java ...). Maybe it should some method to get right position not "opaque position". -- ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment: Thank you for very good explanation. It was hard to understand. I am programming a lot (++10 years) in many language but I still learning new things. -- ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment:
Really really strange but it works :)
"an opaque number when in text mode." -> so it is Windows C libraries.
I use it in production code too so my heart speed up when I see number but it
works as you said.
It looks complicated/slow if I have to open file in binary mode.
I can do it but it is ugly workaround to get position?
It looks some decide that speed is better than functionality so binary files is
only option to get for example estimate progress in some speedometer.
I think that should some function to convert this .tell() for text files into
real .tell().
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
pgn.seek(1008915299)
t = None
while True:
if t:
pgn.seek(t)
pgn.readline()
pt = t
t = pgn.tell()
if pt:
if pt > t:
print('Strange %s!', t)
pgn.seek(t)
print(pgn.tell())
--
___
Python tracker
<https://bugs.python.org/issue39962>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment: Let's test it now. -- ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Change by Cezary Wagner : -- components: +IO, Interpreter Core, Windows nosy: +paul.moore, steve.dower, tim.golden, zach.ware versions: +Python 3.7 ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7)
Cezary Wagner added the comment: I do some test and bu exist in 3.8/3.7 but not in no bug in PyPy3.6/Python2.7. -- title: Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy2.6/Python2.7) -> Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy3.6/Python2.7) ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy2.6/Python2.7)
Change by Cezary Wagner : -- title: Wrong tell function results (Windows 10/Python 64 3.8.2) -> Wrong file.tell() function results (Windows 10/Python 64 3.8.2/3.7 - no bug in PyPy2.6/Python2.7) ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong tell function results (Windows 10/Python 64 3.8.2)
Change by Cezary Wagner : -- title: Wrong tell function results. -> Wrong tell function results (Windows 10/Python 64 3.8.2) ___ Python tracker <https://bugs.python.org/issue39962> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong tell function results.
Cezary Wagner added the comment:
Some good snippet for testing very short.
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
pgn.seek(1008915299)
while True:
pgn.readline()
print(pgn.tell())
1008915327
1008915366
1008915387
1008915409
1008915425
1008915449
1008915471
1008915490
1008915509
1008915534
1008915559
1008915572
1008915631
1008915654
1008915678
1008915680
1008917597
1008917599
1008917631
1008917670
1008917696
1008917718
1008917734
1008917758
1008917780
1008917799
1008917818
1008917843
1008917868
1008917881
1008917942
1008917965
1008917989
1008917991
1008920549
1008920551
1008920583
1008920622
1008920643
1008920663
1008920679
1008920703
1008920725
1008920744
1008920763
1008920788
1008920813
1008920826
1008920877
1008920900
1008920924
1008920926
18446744074718477807 <- ???
1008926192
1008926220
1008926259
1008926276
1008926304
1008926320
1008926344
1008926366
1008926385
1008926404
1008926428
1008926452
1008926465
1008926521
1008926544
1008926568
1008926570
1008928371
1008928373
1008928401
1008928440
1008928460
1008928491
1008928507
1008928531
1008928553
1008928572
1008928591
1008928615
1008928640
1008928653
1008928690
1008928713
1008928737
1008928739
1008931145
1008931147
1008931175
1008931214
1008931233
1008931253
1008931269
1008931293
1008931315
1008931334
1008931353
1008931377
1008931401
1008931414
1008931463
1008931486
1008931516
1008931518
1008937220
1008937222
1008937254
1008937293
1008937315
1008937340
1008937356
1008937380
1008937402
1008937421
1008937440
1008937465
1008937490
1008937503
1008937536
1008937559
18446744074718489200 <- ???
--
___
Python tracker
<https://bugs.python.org/issue39962>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39962] Wrong tell function results.
New submission from Cezary Wagner :
I wrote code which scan very large file PGN (chess games database).
But I found that tell() function is buggy see results.
Here is some code:
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
is_game_parsed = parser.parse_game(visitor=visitor)
# if processing_statistics.games % 100 == 0:
print(processing_statistics.games,
processing_statistics.positions,
processing_statistics.moves,
'%.2f' % processing_statistics.get_games_to_moves(),
'%.2f' % processing_statistics.get_positions_to_moves(),
'%.2f' % speed if speed else speed,
pgn.tell())
print(pgn.tell())
This code can be simplified to this:
with open('../s01_parser_eval/data/out-6976.txt') as pgn:
while True:
pgn.readline()
print(pgn.tell())
1 1 0 0.00 0.00 318.64 1008917597
1008917597
2 47 46 23.00 1.02 343.64 1008917599
1008917599
3 47 46 15.33 1.02 291.08 1008920549
1008920549
4 107 107 26.75 1.00 292.03 1008920551
1008920551
5 107 107 21.40 1.00 185.41 18446744074718477807 <- ???
18446744074718477807
6 234 235 39.17 1.00 157.63 1008926192
1008926192
7 234 235 33.57 1.00 167.75 1008928371
1008928371
8 276 278 34.75 0.99 180.48 1008928373
1008928373
9 276 278 30.89 0.99 185.30 1008931145
1008931145
10 334 336 33.60 0.99 192.58 1008931147
1008931147
11 334 336 30.55 0.99 164.90 1008937220
1008937220
12 468 472 39.33 0.99 149.00 1008937222
1008937222
13 468 472 36.31 0.99 157.58 1008938833
1008938833
14 495 502 35.86 0.99 165.96 1008938835
1008938835
15 495 502 33.47 0.99 167.89 1008941875
1008941875
16 556 567 35.44 0.98 172.10 1008941877
1008941877
17 556 567 33.35 0.98 177.84 1008943769
1008943769
18 591 604 33.56 0.98 184.09 1008943771
1008943771
19 591 604 31.79 0.98 185.38 1008946692
1008946692
20 653 666 33.30 0.98 188.68 1008946694
1008946694
21 653 666 31.71 0.98 192.90 18446744074718500485 <- ???
18446744074718500485
--
messages: 364126
nosy: Cezary.Wagner
priority: normal
severity: normal
status: open
title: Wrong tell function results.
type: crash
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue39962>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35268] Windows 10 asyncio reading continously stdin and stdout Stockfish
Cezary Wagner added the comment: It just tested it today and I found in parallel same solution as you suggested. Sometimes it happen even your skilled programmer - at least one time in year :) I was just blind. Let's close it since it is invalid report. I just skipped new line at end of 'uci' should be 'uci\n'. Now I am unblocked to test async pipes more and write very nice program. It looks very promising. Thank you for attention. I am sorry for bad report. -- ___ Python tracker <https://bugs.python.org/issue35268> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35268] Windows 10 asyncio reading continously stdin and stdout Stockfish
Cezary Wagner added the comment:
Another code try - this time I am using task:
import asyncio
import sys
async def process_line_reader(process, on_line=None, on_eof=None):
while not process.stdout.at_eof():
# BUG? after first line it becomes dead
line = await process.stdout.readline()
if on_line is not None:
on_line(line.decode())
if on_eof is not None:
on_eof()
print('eof')
async def run_stockfish():
STOCKFISH_PATH = r'C:\root\chess\stockfish\stockfish
10\stockfish_10_x64_bmi2.exe'
stockfish = await asyncio.subprocess.create_subprocess_exec(
STOCKFISH_PATH,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stockfish.stdin.write('uci'.encode())
task = asyncio.create_task(process_line_reader(
process=stockfish, on_line=lambda line: print(f'{line}')
))
# await task
await stockfish.wait()
if sys.platform == "win32":
asyncio.set_event_loop_policy(
asyncio.WindowsProactorEventLoopPolicy())
asyncio.run(run_stockfish(), debug=True)
print('done')
All is blocked after first line (no print of eof or done):
C:\root\Python37-64\python.exe "C:/Users/Cezary
Wagner/PycharmProjects/cw_chess_uci/sandbox/async_proxy/s02_async_stockfish.py"
Stockfish 10 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
--
___
Python tracker
<https://bugs.python.org/issue35268>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35268] Windows 10 asyncio reading continously stdin and stdout Stockfish
Cezary Wagner added the comment:
See new code example (little changed):
It block after first stdout:
import asyncio
import sys
async def run_stockfish():
STOCKFISH_PATH = r'C:\root\chess\stockfish\stockfish
10\stockfish_10_x64_bmi2.exe'
stockfish = await asyncio.subprocess.create_subprocess_exec(
STOCKFISH_PATH,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stockfish.stdin.write('uci'.encode())
while not stockfish.stdout.at_eof():
# BUG? - blocks at this line
line = await stockfish.stdout.readline()
print(line.decode())
await stockfish.wait()
if sys.platform == "win32":
asyncio.set_event_loop_policy(
asyncio.WindowsProactorEventLoopPolicy())
asyncio.run(run_stockfish(), debug=True)
Output:
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit
(AMD64)] on win32
runfile('C:/Users/Cezary
Wagner/PycharmProjects/cw_chess_uci/sandbox/async_proxy/s01_async_stockfish.py',
wdir='C:/Users/Cezary Wagner/PycharmProjects/cw_chess_uci/sandbox/async_proxy')
Stockfish 10 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
Valid output is:
C:\Users\Cezary Wagner>"C:\root\chess\stockfish\stockfish
10\stockfish_10_x64_bmi2.exe"
Stockfish 10 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
uci
id name Stockfish 10 64 BMI2
id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott
option name Debug Log File type string default
option name Contempt type spin default 24 min -100 max 100
option name Analysis Contempt type combo default Both var Off var White var
Black var Both
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 131072
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 30 min 0 max 5000
option name Minimum Thinking Time type spin default 20 min 0 max 5000
option name Slow Mover type spin default 84 min 10 max 1000
option name nodestime type spin default 0 min 0 max 1
option name UCI_Chess960 type check default false
option name UCI_AnalyseMode type check default false
option name SyzygyPath type string default
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 7 min 0 max 7
uciok
--
___
Python tracker
<https://bugs.python.org/issue35268>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35268] Windows 10 asyncio reading continously stdin and stdout Stockfish
Cezary Wagner added the comment: Stockfish works like that: 1. Run command line and listen stdout. 2. Send some stdin and listen stdin. 3. stdout is asynchronous and continous (there no single output, line is generated every some time). I want to send some stdin than listen stdout response from Stockfish. I tested that first execute of it block all: line = await stockfish.stdout.read() One line is read and application becomes dead. Should show much more lines but it stops in Windows 10. -- title: Windows asyncio reading continously stdin and stdout Stockfish -> Windows 10 asyncio reading continously stdin and stdout Stockfish ___ Python tracker <https://bugs.python.org/issue35268> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35268] Windows asyncio reading continously stdin and stdout Stockfish
Cezary Wagner added the comment: It looks like StremReader.read() not work and StremReader.readline() in Windows or I am not understand documentation. https://docs.python.org/3/library/asyncio-stream.html coroutine read(n=-1) Read up to n bytes. If n is not provided, or set to -1, read until EOF and return all read bytes. If EOF was received and the internal buffer is empty, return an empty bytes object. coroutine readline() Read one line, where “line” is a sequence of bytes ending with \n. If EOF is received and \n was not found, the method returns partially read data. If EOF is received and the internal buffer is empty, return an empty bytes object. -- ___ Python tracker <https://bugs.python.org/issue35268> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35268] Windows asyncio reading continously stdin and stdout Stockfish
New submission from Cezary Wagner :
I have some problems with asyncio on Windows - it block where it should go.
Documentation shows that it should work:
https://docs.python.org/3/library/asyncio-stream.html -> StreamReader should
return something.
See 1st example:
import asyncio
import sys
async def run_stockfish():
STOCKFISH_PATH = r'C:\root\chess\stockfish\stockfish
9\stockfish_9_x64_bmi2.exe'
stockfish = await asyncio.subprocess.create_subprocess_exec(
STOCKFISH_PATH,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stockfish.stdin.write('uci'.encode())
while True:
# BUG? - blocks at this line - no output
line = await stockfish.stdout.read()
print(line)
await stockfish.wait()
if sys.platform == "win32":
asyncio.set_event_loop_policy(
asyncio.WindowsProactorEventLoopPolicy())
asyncio.run(run_stockfish(), debug=True)
1st output (nothing):
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit
(AMD64)] on win32
runfile('C:/Users/Cezary
Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy/s01_async_stockfish.py',
wdir='C:/Users/Cezary
Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy')
2nd example:
import asyncio
import sys
async def run_stockfish():
STOCKFISH_PATH = r'C:\root\chess\stockfish\stockfish
9\stockfish_9_x64_bmi2.exe'
stockfish = await asyncio.subprocess.create_subprocess_exec(
STOCKFISH_PATH,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stockfish.stdin.write('uci'.encode())
while True:
# BUG? - blocks at this line
line = await stockfish.stdout.readline()
print(line)
await stockfish.wait()
if sys.platform == "win32":
asyncio.set_event_loop_policy(
asyncio.WindowsProactorEventLoopPolicy())
asyncio.run(run_stockfish(), debug=True)
2nd output is little better (first line is read):
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit
(AMD64)] on win32
runfile('C:/Users/Cezary
Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy/s01_async_stockfish.py',
wdir='C:/Users/Cezary
Wagner/PycharmProjects/stockfish-proxy/sandbox/async_proxy')
b'Stockfish 9 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott\r\n'
What should be done in both case (or maybe done):
Stockfish 9 64 BMI2 by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
uci
id name Stockfish 9 64 BMI2
id author T. Romstad, M. Costalba, J. Kiiski, G. Linscott
option name Debug Log File type string default
option name Contempt type spin default 20 min -100 max 100
option name Threads type spin default 1 min 1 max 512
option name Hash type spin default 16 min 1 max 131072
option name Clear Hash type button
option name Ponder type check default false
option name MultiPV type spin default 1 min 1 max 500
option name Skill Level type spin default 20 min 0 max 20
option name Move Overhead type spin default 30 min 0 max 5000
option name Minimum Thinking Time type spin default 20 min 0 max 5000
option name Slow Mover type spin default 89 min 10 max 1000
option name nodestime type spin default 0 min 0 max 1
option name UCI_Chess960 type check default false
option name SyzygyPath type string default
option name SyzygyProbeDepth type spin default 1 min 1 max 100
option name Syzygy50MoveRule type check default true
option name SyzygyProbeLimit type spin default 6 min 0 max 6
uciok
--
components: Windows, asyncio
messages: 330028
nosy: Cezary.Wagner, asvetlov, paul.moore, steve.dower, tim.golden, yselivanov,
zach.ware
priority: normal
severity: normal
status: open
title: Windows asyncio reading continously stdin and stdout Stockfish
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue35268>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23514] multiprocessing documentation - little more examples for parallel computing
Changes by Cezary Wagner cezary.wag...@gmail.com: -- versions: +Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23514 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23510] multiprocessing bug SyncManager and 'with'
Cezary Wagner added the comment: Yes it is what is wanted by me and other people I think. 1. enter has not start call. 2. exit has shutdown but it is not supported 3. 'with' is what is need since it is simple to manage Good analysis. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23510 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23513] Add support for classes/object model in multiprocessing/pickle
Cezary Wagner added the comment: I am fighting with multiprocessing and still not won - I need to invent new Python coding style to use it and can not reduce ugly code/microcoding. It is not because it is not possible to do it in Python easier but because it is not improved yet. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23513 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23514] multiprocessing documentation - little more examples for parallel computing
Cezary Wagner added the comment: Should I add this suggestion to open before issue? I started doing multiprocessing code in Python so I can have more suggestions since I found that documentation can be improved I will report specific problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23514 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
