Марк Коренберг added the comment:
The same in tarfile.copyfileobj()
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26877>
___
___
New submission from Марк Коренберг:
tarfile.py: _FileInFile():
(near line 687)
b = self.fileobj.read(length)
if len(b) != length:
raise ReadError("unexpected end of data")
every read() API does not guarantee that it will read `length` bytes. So, if
fileobj reads less than
Марк Коренберг added the comment:
https://github.com/SpiderOak/ZipStream tries to implement streaming in one more
kind.
Also libarchive have experimental support of streaming write to zip archives in
newest version.
So problem is actual
Марк Коренберг added the comment:
Also, Python have problems with streaming READ of zip archive. I mean ability
to read (in some form iterate over) archive when seeking is not available.
I mean iteration like one in TAR archives.
--
___
Python
Марк Коренберг added the comment:
I have the same problem, and make monkey-patch by myself BEFORE seeing this
issue (!)
Example how I can do that is attached under name "socketpair.py".
It will be nice if you take my idea. And after that streaming of zip files
would b
Марк Коренберг added the comment:
In a common case,
if (write(thread.fd, thread.header, thread.header_len) == -1)
should be replaced with
if (write(thread.fd, thread.header, thread.header_len) != thread.header_len)
--
nosy: +mmarkk
___
Python
Марк Коренберг added the comment:
$ python3.5
Python 3.5.0+ (default, Oct 11 2015, 09:05:38)
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> u'a\nb£
Марк Коренберг added the comment:
No, I have checked, it returns list of `bytes` objects.
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
in order to increase perofrmance even more, use block operation on bytes. I.e.
Xor by 8 bytes first (on 64-bit system) while size remainig is bigger or equal
to 8, then by 4 bytes using same loop, and then xor remaining bytes by one
byte. This will increase
Марк Коренберг added the comment:
The main idea: if wile does not have a name, it should not have it!
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
1. nasty bugs not in my code!
2. It is not documented that name may be an int, so if applications rely on
undocumented stuff are definitely broken, so if IMHO we CAN change that.
3. It will be much stricter to delattr('name') instead of setting as None
Марк Коренберг added the comment:
oops, issue22208 is not related issue
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26318>
___
__
Марк Коренберг added the comment:
You should add a tests. especially for edge cases, for negative values for
example.
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
In particular, `io.open(fd, ...)` can not be used with tarfile, since it use
it's "name" property, and think that it is real file name, and not expecte it
to be int.
--
___
Python tracker <rep...@bu
Марк Коренберг added the comment:
in any case, passing INTEGER as name is wrong thing, since (due to duck
typing), name should be a string.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
New submission from Марк Коренберг:
`io.open(fd, ...).name` returns numeric fd instead of None. This lead to some
nasty bugs.
In order to bring consistency and make that predictable, please make `.name`
for that case to return None. (and document it)
--
components: IO, Library (Lib
New submission from Марк Коренберг:
This works right in Python 2.7, but fails in python3:
UnboundLocalError: local variable 'e' referenced before assignment
def test():
try:
raise Exception('a')
except Exception as e:
pass
else:
return
print(e)
test
Марк Коренберг added the comment:
text files and seek() offset: issue25849
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
During development, we decide not to change readline() function. So, yes, this
issue is closed.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Changes by Марк Коренберг <socketp...@gmail.com>:
--
keywords: +patch
Added file:
http://bugs.python.org/file41595/asyncio-stream-doc-preliminary.patch
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Марк Коренберг added the comment:
Attached patch, but did not compile documentation until text is validated.
These descriptions are just copies of docstrings of corresponding functions.
--
___
Python tracker <rep...@bugs.python.org>
New submission from Марк Коренберг:
See code discussion here:
https://github.com/python/asyncio/pull/297
--
components: asyncio
messages: 257776
nosy: gvanrossum, haypo, mmarkk, yselivanov
priority: normal
severity: normal
status: open
title: Add new StreamReader.readuntil() method
Марк Коренберг added the comment:
16.2.4.3. Multi-threading
FileIO objects are thread-safe to the extent that the operating system calls
(such as read(2) under Unix) they wrap are thread-safe too.
Binary buffered objects (instances of BufferedReader, BufferedWriter,
BufferedRandom
Марк Коренберг added the comment:
sys.stdin
sys.stdout
sys.stderr
...
These streams are regular text files
...
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Just mention it here :)
https://github.com/KeepSafe/aiohttp/issues/686
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Changes by Марк Коренберг <socketp...@gmail.com>:
--
resolution: -> wont fix
status: open -> closed
___
Python tracker <rep...@bugs.python.org>
<http://bugs.
Марк Коренберг added the comment:
Yes, can I do it on, say, github ?
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25927>
___
__
Марк Коренберг added the comment:
instead of realpath, os.lstat() may be used in order to detect if target file
object is not plain file. Also, os.rename() cannot be used since it follows
symlinks by default.
--
___
Python tracker <
Марк Коренберг added the comment:
Also, modern kernels allows to replace entire directory!
renameat2() + RENAME_EXCHANGE
So, there should be more atomic operations
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/
Марк Коренберг added the comment:
David Murray, in your code, if temporary file cannot be created, it attempts to
unlink unknown name :).
But, I already have almost the same solution in production. I mention it
sources in issue8604.
--
___
Python
New submission from Марк Коренберг:
dir_fd support is good in `os` module. but tempfile module lack support of that
great feature. Please add.
--
messages: 256856
nosy: mmarkk
priority: normal
severity: normal
status: open
title: add dir_fd for mkstemp, and also maybe to all tempfile
Марк Коренберг added the comment:
You also forgot about two things:
1. set temporary file permissions before rename
2. fsync(open(os.dirname(temporaryfile)))
3. if original file name is symlink, replace will works wrong. os.realpath
should be used.
So, here are real life function, that we use
Марк Коренберг added the comment:
But... os.openat() is still missing... why status is closed() ?!
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Changes by Марк Коренберг <socketp...@gmail.com>:
--
components: +Library (Lib)
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python
Марк Коренберг added the comment:
So, I'm required to re-implement NamedTemporaryFile by hand like that:
fd, name = tempfile.mkstemp(...)
try:
with io.open(fd, ...) as fff:
fff.write('hello')
fff.flush()
os.fdatasync(fff
Марк Коренберг added the comment:
ReadStream.read_until() has been implemented instead enhancing readline().
--
resolution: -> wont fix
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.or
Марк Коренберг added the comment:
ping. THis is essential in our software, so we use private field in order to
work-around this bug.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Really optimal search algorithm should be something like that:
https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.p
Марк Коренберг added the comment:
Well, 03e61104f7a2 adds good description, why not to enforce checks instead of
saying that some values are unsupported ?
Also, idea in returning special object instance from tell(), this object should
incapsulate byte offset. And allow for the seek() either
Марк Коренберг added the comment:
First, it seems that there are no real "reconstruction algorithm" at all. Seek
is allowed to point to any byte position, even to place "inside" characters for
multibyte encodings, such as UTF-8.
Second, about performance: I talk
Марк Коренберг added the comment:
s/peek/tell/
--
status: closed -> open
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25849>
___
___
Марк Коренберг added the comment:
Also, can you provide the case, where such random seeks can be used on text
files? It would be programmer error to seek to places other I mention. Does not
it ?
--
___
Python tracker <rep...@bugs.python.org>
Марк Коренберг added the comment:
https://docs.python.org/3.5/library/io.html?highlight=stringio#id3 :
Also, TextIOWrapper.tell() and TextIOWrapper.seek() are both quite slow due to
the reconstruction algorithm used.
What is reconstruction algorightm ? Experiments show, that seek() and tell
Марк Коренберг added the comment:
note, that there are many other usecases for that function, like chattr, chmod
g+w, touch and so on.
But I'm personally consider this will bloat python library, since everyone can
write it's own cycle over os.walk() in his program.
Also, chown itself
Марк Коренберг added the comment:
Instead, it may be desirable to implement wrapper over ow.walk() that apply
given function to every member.
i.e.:
recursive_apply('/some/dir', lambda x: os.chown(x, 42, 42))
--
nosy: +mmarkk
___
Python tracker
New submission from Марк Коренберг:
It seems, that we should deprecate .seek() on files, opened in text mode.
Since it is not possible to seek to position between symbols. Yes, it is
possible to decode UTF-8 (or other charset) starting from beginning of the file
and count symbols
Марк Коренберг added the comment:
#25849 ?
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue25190>
___
___
Pyth
Марк Коренберг added the comment:
Okay, https://github.com/python/asyncio/pull/298 is ready
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Yes, this is not a bug. Python 3.5 works as expected.
Orderable lambda is the bug in python2.7.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Well, https://github.com/python/asyncio/pull/297 is ready :)
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Please close this bug since asyncio is much more suitable, and things like
socketserver is obsolete as I think.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
When I reported that bug, I want to suggest multibyte separators, but
implementation is much complex than current one. For example, we should decide
what to do if partial separator is read and EOF occur.
So, I think we should merge ability to use custom one
Марк Коренберг added the comment:
Do not understand what should be done in order to fix that...
It will be nice if I can create pull request by myself
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Yes this is not a bug.
Use asyncio.gather(), it preserves order of elements :)
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Changes by Марк Коренберг <socketp...@gmail.com>:
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue15500>
___
_
Марк Коренберг added the comment:
result of readexactly(0) is indistinguishable from EOF, so no one should try to
use that as I think.
What is the useful case when zero bytes should be read ?
--
___
Python tracker <rep...@bugs.python.org>
New submission from Марк Коренберг:
Here is the match against master.
Doc/library/asyncio-protocol.rst:
@@ -156,9 +156,9 @@ WriteTransport
high-water limit is given, the low-water limit defaults to an
implementation-specific value less than or equal to the
high-water
Марк Коренберг added the comment:
No activity last week ? Why not to merge ?
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Please merge patch, proposed by Eduardo Seabra (Eduardo.Seabra). And document
that this is UNIX-only solution.
--
versions: +Python 3.5, Python 3.6
Added file: http://bugs.python.org/file41055/test_tempfile.py
New submission from Марк Коренберг:
ascynio readexactly() should raise ValueError if passed length <= 0 in argument.
Now, it return empty string, which is just error hiding. Why not to raise
ValueError ?
Returning empty string is error prone. This behaviour was not changed since
init
Марк Коренберг added the comment:
Bug still reproduced. Jenkins running from init.d use /usr/bin/daemon. This
mean SIGHUP will be in SIG_IGN state. Since echo.py does not setup sighup
handler, sighup will be equivalent of SIGKILL. So, why not to use, say, SIGTERM
instead? After such change
Марк Коренберг added the comment:
Man getrandom()
As of Linux 3.19, the following bug exists:
* Depending on CPU load, getrandom() does not react to interrupts
before reading all bytes requested.
So, is it goot to use this syscall now
Марк Коренберг added the comment:
Okay, seemes it is not documented that
os.open('.', os.O_RDWR |os.O_EXCL | os.O_DIRECTORY)
Should return EISDIR
I did not found that in Linux manpages. Using undocumented features is bad.
Maybe I should report this to Michael Kerrisk to update manpage
Марк Коренберг added the comment:
Well, it's not said explicit, that O_DIRECTORY cannot be combined with O_RDWR.
So, everything is valid now, very hacky, but works without bugs.
It will be nice, if someone comment that hacks in source code
Марк Коренберг added the comment:
Huge thanks for that patch. Now things are much cleaner.
--
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Just for link. Issue #21515 — tempfile use this functionality now.
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Suppose conditions:
- Old linux kernel ignoring flag
- malicious hacker force use of PLAIN FILE instead of directory
On new kernel it will fail
On old kernel it will just open that file!
So, we can make a HACK! Just add last slash to directory name
Марк Коренберг added the comment:
Just install rngd and setup it to user /dev/urandom as entropy source. I think
thread is closed :)
--
nosy: +mmarkk
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/i
Марк Коренберг added the comment:
Please reopen. I still have interest for that. Fact that we survive 8 years is
not enough for closing bugs as "Won't fix".
This fact point only on that ther are no good specialists that can fix that
bug, since it is Windows-only
Марк Коренберг added the comment:
Sorry, for the "there are no good specialist". I mean "There are no well
motivated good specialist" :)
--
___
Python tracker <rep...@bugs.python.org>
<http
New submission from Марк Коренберг:
import heapq
from functools import partial
qwe = [(0, lambda x: 42), (0, lambda x: 56)]
heapq.heapify(qwe)
qwe = [(0, partial(lambda x: 42)), (0, partial(lambda x: 56))]
heapq.heapify(qwe)
Traceback (most recent call last):
File /usr/lib/python3.4
Марк Коренберг added the comment:
Exactly the same with bound class methods
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24660
___
___
Python
New submission from Марк Коренберг:
Suppose that program:
import asyncio
import socket
def receiver(loop):
(a, b) = socket.socketpair()
loop.call_later(1, lambda: print('Should be called inside the loop'))
end = loop.time() + 3
print
Марк Коренберг added the comment:
$ PYTHONASYNCIODEBUG=1 ./bug.py
Starting busy receiver
Busy receiver complete
Executing Task pending coro=receiver() running at ./bug.py:16
wait_for=Future pending cb=[Task._wakeup()] created at
/usr/lib/python3.4/asyncio/tasks.py:490 cb
Марк Коренберг added the comment:
Adding of b.setblocking(0) after socketpair() does not help.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue24532
New submission from Марк Коренберг:
If I pass an list of futures/coroutines as (done, _) = asyncio.wait([...]),
order of elements in `done` does NOT match order of elements in `wait`'s
argument. This is not documented, and I don't know if that is a bug or not.
Also, documentation say:
Wait
Changes by Марк Коренберг socketp...@gmail.com:
--
assignee: - docs@python
components: +Documentation, asyncio
nosy: +docs@python, gvanrossum, haypo, yselivanov
type: - enhancement
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
New submission from Марк Коренберг:
These function returns handle, so, registered callback can be cancelled.
--
messages: 245803
nosy: mmarkk
priority: normal
severity: normal
status: open
title: Undocumented features of asyncio: call_at, call_later
versions: Python 3.6
Марк Коренберг added the comment:
For delayed execution methods:
Returned handle is actually timer object. That timer can be deactivated using
asyncio.Handle.cancel() method, so registered callback won't be called.
For other callback registration methods:
Returned handle may be used
Марк Коренберг added the comment:
So, maybe API change? like delete=True|False|Maybe ? don't think that this is
good decisions.
My approach is based on ext4 behaviour about delayed allocation and atomic file
replacements.
In my case, either old file (with contents) or new file appear.
In any
Марк Коренберг added the comment:
why not to make tmpfileobj.delete as property that really sets
self._closer.delete ? this will fix my problem.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21579
Марк Коренберг added the comment:
Is issue 21579 fixed in that bug? too many letters..sorry...
--
nosy: +mmarkk
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18879
Марк Коренберг added the comment:
Yes, but O_TMPFILE should be set ONLY when used with TemporaryFile, not with
NamedTemporaryFile. My problem refer only NamedTemporaryFile.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
New submission from Марк Коренберг:
Suppose code:
=
import os
import tempfile
want_to_replace = 'zxc.dat'
tmpdir=os.path.dirname(os.path.realpath(want_to_replace))
with tempfile.NamedTemporaryFile(dir=tmpdir) as fff:
# do somewhat with fff here
Марк Коренберг added the comment:
Calling getdents()/readdir64() repeatedly while closing descriptors provides
unexpected behaviour. Reading directory while it modified is not safe by
default. For example: http://en.it-usenet.org/thread/18514/15719/.
So, we should re-open directory if we
Марк Коренберг added the comment:
Also, it is not said in manual if getdents() may be interrupted by signal.
Assuming current code, error is not checked, so some (or all) descriptors will
be skipped in case of error.
--
___
Python tracker rep
New submission from Марк Коренберг:
1. Please see last comments/patches for issue8052
2. Not closing some descriptos is security breach (PEP-0446 describes that)
=
Calling getdents()/readdir64() repeatedly while closing descriptors provides
unexpected behaviour. Reading
New submission from Марк Коренберг:
Results or running attached program:
$ python2.7 qwe.py
TCP mode, makefile method. 198807.2 lines per second (189.6 MB/s). Delay is
5.03 seconds
TCP mode, fdopen method. 1041666.7 lines per second (993.4 MB/s). Delay is
0.96 seconds
UNIX mode, makefile
Марк Коренберг added the comment:
Yes, results are repeatable, and for python 2.7 I have roughly same timings for
UNIX socket.
Also, I have straced all variants and see that in all 4 cases (and for both
python versions) IO is done using 8192 blocks in size, so buffering is not
cause
Марк Коренберг added the comment:
Well, python 3.3 is slightly faster:
$ python3.3 qwe.py
TCP mode, makefile method. 380228.1 lines per second (362.6 MB/s). Delay is
2.63 seconds
TCP mode, fdopen method. 877193.0 lines per second (836.6 MB/s). Delay is
1.14 seconds
UNIX mode, makefile
Changes by Марк Коренберг socketp...@gmail.com:
Removed file: http://bugs.python.org/file30731/qwe.py
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18329
Марк Коренберг added the comment:
Eliminate unicode conversion for python3, but results still the same
$ python2.7 qwe.py
TCP mode, makefile method. 211416.5 lines per second (201.6 MB/s). Delay is
4.73 seconds
TCP mode, fdopen method. 1041666.7 lines per second (993.4 MB/s). Delay
Марк Коренберг added the comment:
Can anyone test in python 3.4 ?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18329
___
___
Python-bugs-list
New submission from Марк Коренберг:
Today, subprocess allow to change directory only by using its name. What if I
have only file descriptor referring to that dir?
It will be nice if such feture will be implemented. Now, I use preexc_fn to
call os.fchdir() by hand.
We should handle close_fds
New submission from Марк Коренберг:
In linux (Since 2.6.9) we can use syscall
prctl(PR_SET_NAME, Some thread name)
to set thread name to the kernel. This thread is seen under this name in some
process tool (like top, ps, pstree (have bug reported connected with this) and
others
Changes by Марк Коренберг socketp...@gmail.com:
--
title: Set thread nema in linux kernel - Set thread name in linux kernel
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18006
Марк Коренберг added the comment:
Ups. hiding EOFHeaderError is not an error.
But handilng of other errors is not perfect. Please review TarFile.next() for
cases where .tar file is corrupted. For example,
TruncatedHeaderError is re-raised only if problem at the start of the file.
Really
Марк Коренберг added the comment:
Yes, re-writing windows IO to direct API, without intemediate layer is still
needed.
Please don't close bug. Maybe someone will implement this.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org
Марк Коренберг added the comment:
Yes, bug exists in python 2.7. The same problem - negative timestamp in mtime
field. It prepresented in binary value as
'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xc7\xfc'
in my case.
--
nosy: +mmarkk
___
Python
Марк Коренберг added the comment:
And yes, bug does not appear in python3.2
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14810
___
___
Python
201 - 300 of 385 matches
Mail list logo
