Re: Make a unique filesystem path, without creating the file
Marko Rauhamaa writes: > It is also correct that /dev/urandom depletes the entropy pool as > effectively as /dev/random. I think see what's confusing you: the above is a misconception that is probably held by lots of people. Entropy is not water and from a cryptographic standpoint there is essentially no such thing as "depleting" an entropy pool. There is either enough entropy (say 256 bits or more) in the PRNG or else there isn't. If there's not enough, urandom can misbehave by giving you bad output because it doesn't block until more is gathered. If there is enough, /dev/random misbehaves by blocking under this bogus concept of "depletion". If you have a seed with 256 bits of entropy and you generate a gigabyte of random numbers from it, you have not increased the predictability of the seed in any significant way. So once /dev/random unblocks, it should never again block, the behavior of getrandom. There used to be an article on David Wagner's web site (cs.berkeley.edu/~daw) about the concept of "depleting" entropy by iterated hashing, but I can't find it now. That's unfortunate since it might help cast light on the subject. >> http://www.2uo.de/myths-about-urandom/ > Already addressed. No really, all you've done is repeat bad advice. The people cited in that article are very knowledgeable and the stuff they say makes good mathematical sense. The stuff you say makes no sense and you haven't given any convincing reason for anyone to listen to you. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Steven D'Aprano : > On Tue, 23 Feb 2016 06:32 am, Marko Rauhamaa wrote: >> Under Linux, /dev/random is the way to go when strong security is >> needed. Note that /dev/random is a scarce resource on ordinary >> systems. > > That's actually incorrect, but you're not the only one to have been > mislead by the man pages. > > http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/ Still, mostly hypnotic repetitions. However, it admits: But /dev/random also tries to keep track of how much entropy remains in its kernel pool, and will occasionally go on strike if it decides not enough remains. That's the whole point. /dev/random will rather block the program than lower the quality of the random numbers below a threshold. /dev/urandom has no such qualms. If you use /dev/random instead of urandom, your program will unpredictably (or, if you’re an attacker, very predictably) hang when Linux gets confused about how its own RNG works. Yes, possibly indefinitely, too. Using /dev/random will make your programs less stable, but it won’t make them any more cryptographically safe. It is correct that you shouldn't use /dev/random as a routine source of bulk random numbers. It is also correct that /dev/urandom depletes the entropy pool as effectively as /dev/random. However, when you are generating signing or encryption keys, you should use /dev/random. As stated in https://lwn.net/Articles/606141/>: /dev/urandom should be used for essentially all random numbers required, but /dev/random is sometimes used for things like extremely sensitive, long-lived keys (e.g. GPG) or one-time pads. > See also: > > http://www.2uo.de/myths-about-urandom/ Already addressed. Marko -- https://mail.python.org/mailman/listinfo/python-list
good python tutorial
This site was recommended by a friend, it looks really well put together, I thought it might be of interest to people considering online tutorials. http://www.python-course.eu/index.php -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Chris Angelico writes: > How much future are you expecting? This is old but its methodology still seems ok: http://saluc.engr.uconn.edu/refs/keymgr/blaze95minimalkeylength.pdf I also like this: http://cr.yp.to/talks/2015.10.05/slides-djb-20151005-a4.pdf Quote (slide 37): The crypto users' fantasy is boring crypto: crypto that simply works, solidly resists attacks, never needs any upgrades. HN discussion: https://news.ycombinator.com/item?id=10345965 -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 1:27 PM, Paul Rubin wrote: > 3) The default token length should be long enough to not have to "change > in the future". If the user wants a shorter token, they ask for that, > or can truncate a longer one that they receive from the default. How much future are you expecting? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Steven D'Aprano writes: > https://www.python.org/dev/peps/pep-0506/ I didn't know about this! The discussion was all on mailing lists? A few things I suggest changing: 1) the default system RNG for Linux should be getrandom(2) on kernels that support it (3.17 and later). 2) Some effort should be directed at simulating getrandom's behaviour on kernels that don't have it, using the /dev/random entropy estimator and the /dev/urandom interface. I.e. it should block if the system hasn't seen enough entropy to get the CSPRNG started securely, and never block after that. 3) The default token length should be long enough to not have to "change in the future". If the user wants a shorter token, they ask for that, or can truncate a longer one that they receive from the default. There are a few other choices in the PEP whose benefit is unclear to me, but they aren't harmful, and I guess the decisions have already been made. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-23, Ben Finney wrote: > Oscar Benjamin writes: >> What does unpredictable mean in this context? Maybe I'm reading too >> much into that... > > I think you may be, yes. The request in this thread requires making > direct use of the “generate a new valid temporary fielsystem path” > functionality already implemented in ‘tempfile’. > > Implementations of that functionality outside of ‘tempfile’ are a fun > exercise, but miss the point of this thread. I think you have missed the point of your own thread. You can't do what you wanted using tempfile, the only possible answer is to choose a filename that is sufficiently random that your hope that it is unique won't be proven futile. tempfile has two main modes, mktemp which meets your requirements but should never be used as it is insecure, and mkstemp which doesn't meet your requirements because it fundamentally operates by actually creating the file in question and relying on the filesystem to guarantee uniqueness. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 11:44 AM, Jon Ribbens wrote: > On 2016-02-23, Chris Angelico wrote: >> On Tue, Feb 23, 2016 at 11:26 AM, Jon Ribbens >> wrote: >>> On 2016-02-23, Chris Angelico wrote: On Tue, Feb 23, 2016 at 11:08 AM, Jon Ribbens wrote: >> If you generate 2**128 + 1 such numbers, you are *guaranteed* to > > ... have expired due to the heat death of the universe. Maybe... but by the time you get to 2**64 of them, you have a 50% chance of a collision. (That's either utterly intuitive or completely counter-intuitive, depending on who you are.) >>> >>> Um, did you mean to say 2**127? Are you thinking of the >>> birthday paradox or something, which doesn't apply here? >> >> By the time you generate 2**64 of them, you have a 50% chance that >> some pair of them collides. Yes, the birthday paradox does apply here. > > Oh, I see, you're thinking of it differently. I was thinking of it as > Alice is choosing a filename and Mallet is trying to guess it, in which > case the birthday paradox doesn't apply. You're thinking of it as Alice > is generating many random filenames and, even though she could avoid > collisions with 100% certainty by remembering what she's already had, > isn't doing so, and must avoid colliding with herself. I don't think > your version makes has much relevance as an attack model. Ah. Steven was talking about collisions; once you have 2**128+1 of them, you're guaranteed a collision (pigeonhole principle). What you're talking about gives certainty slightly sooner - specifically, once you've tried 2**128 of them, you're guaranteed to have hit it :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-23, Chris Angelico wrote: > On Tue, Feb 23, 2016 at 11:26 AM, Jon Ribbens > wrote: >> On 2016-02-23, Chris Angelico wrote: >>> On Tue, Feb 23, 2016 at 11:08 AM, Jon Ribbens >>> wrote: > If you generate 2**128 + 1 such numbers, you are *guaranteed* to ... have expired due to the heat death of the universe. >>> >>> Maybe... but by the time you get to 2**64 of them, you have a 50% >>> chance of a collision. (That's either utterly intuitive or completely >>> counter-intuitive, depending on who you are.) >> >> Um, did you mean to say 2**127? Are you thinking of the >> birthday paradox or something, which doesn't apply here? > > By the time you generate 2**64 of them, you have a 50% chance that > some pair of them collides. Yes, the birthday paradox does apply here. Oh, I see, you're thinking of it differently. I was thinking of it as Alice is choosing a filename and Mallet is trying to guess it, in which case the birthday paradox doesn't apply. You're thinking of it as Alice is generating many random filenames and, even though she could avoid collisions with 100% certainty by remembering what she's already had, isn't doing so, and must avoid colliding with herself. I don't think your version makes has much relevance as an attack model. -- https://mail.python.org/mailman/listinfo/python-list
Re: How the heck does async/await work in Python 3.5
On Mon, Feb 22, 2016 at 3:16 PM, Sven R. Kunze wrote: > Is something like shown in 12:50 ( cout << tcp_reader(1000).get() ) possible > with asyncio? (tcp_reader would be async def) loop = asyncio.get_event_loop() print(loop.run_until_complete(tcp_reader(1000))) -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, 23 Feb 2016 06:32 am, Marko Rauhamaa wrote:
> Jon Ribbens :
>
>> Suppose you had code like this:
>>
>> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>>
>> Do we really think that is insecure or that there are any practical
>> attacks against it? It would be basically the same as saying that
>> urandom() is broken, surely?
>
> urandom() is not quite random and so should not be considered
> cryptographically airtight.
>
> Under Linux, /dev/random is the way to go when strong security is
> needed. Note that /dev/random is a scarce resource on ordinary systems.
That's actually incorrect, but you're not the only one to have been mislead
by the man pages.
http://sockpuppet.org/blog/2014/02/25/safely-generate-random-numbers/
On non-Linux Unixes, the difference between urandom and random is mostly, or
entirely, gone, in favour of urandom's non-blocking behaviour. And it's a
myth that the output of random is "more random" or "more pure" than
urandom's. In reality, on Linux both urandom and random use exactly the
same CSPRNG.
See also:
http://www.2uo.de/myths-about-urandom/
for a good explanation of how random and urandom actually work on Linux.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 11:26 AM, Jon Ribbens wrote: > On 2016-02-23, Chris Angelico wrote: >> On Tue, Feb 23, 2016 at 11:08 AM, Jon Ribbens >> wrote: If you generate 2**128 + 1 such numbers, you are *guaranteed* to >>> >>> ... have expired due to the heat death of the universe. >> >> Maybe... but by the time you get to 2**64 of them, you have a 50% >> chance of a collision. (That's either utterly intuitive or completely >> counter-intuitive, depending on who you are.) > > Um, did you mean to say 2**127? Are you thinking of the > birthday paradox or something, which doesn't apply here? By the time you generate 2**64 of them, you have a 50% chance that some pair of them collides. Yes, the birthday paradox does apply here. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Oscar Benjamin writes: > What does unpredictable mean in this context? Maybe I'm reading too > much into that... I think you may be, yes. The request in this thread requires making direct use of the “generate a new valid temporary fielsystem path” functionality already implemented in ‘tempfile’. Implementations of that functionality outside of ‘tempfile’ are a fun exercise, but miss the point of this thread. -- \ “But Marge, what if we chose the wrong religion? Each week we | `\ just make God madder and madder.” —Homer, _The Simpsons_ | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-23, Chris Angelico wrote: > On Tue, Feb 23, 2016 at 11:08 AM, Jon Ribbens > wrote: >>> If you generate 2**128 + 1 such numbers, you are *guaranteed* to >> >> ... have expired due to the heat death of the universe. > > Maybe... but by the time you get to 2**64 of them, you have a 50% > chance of a collision. (That's either utterly intuitive or completely > counter-intuitive, depending on who you are.) Um, did you mean to say 2**127? Are you thinking of the birthday paradox or something, which doesn't apply here? -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, 23 Feb 2016 05:17 am, Jon Ribbens wrote:
> On 2016-02-22, Ethan Furman wrote:
>> On 02/14/2016 04:08 PM, Ben Finney wrote:
>>> I am unconcerned with whether there is a real filesystem entry of that
>>> name; the goal entails having no filesystem activity for this. I want a
>>> valid unique filesystem path, without touching the filesystem.
>>
>> This is impossible. If you don't touch the file system you have no way
>> to know if the path is unique.
>
> Weell, I have a lot of sympathy for that point, but on the other
> hand the whole concept of UUIDs ("import uuid") is predicated on the
> opposite assumption.
You're referring to uuid4, presumably, as the other varieties of UUID use
non-secret information, such as the time, or a namespace, either of which
is potentially public knowledge.
Only uuid4 is considered "globally unique", and that's not *certainly*
globally unique, only that the chances of an *accidental* collision is
below some threshold deemed "small enough that we don't care".
Deliberate collisions of public UUIDs are *trivial*. Pick a UUID you know is
already in use, and use it again.
There's a lot of assumptions involved in the "globally unique" claim, and
there are probably ways to contrive to generate the same UUIDs as someone
else. But to what benefit? UUIDs are not intended as security tokens, and
are not hardened against attack. Even uuid4 may not be suitable for
security, since it may use a cryptographically weak PRNG such as Mersenne
Twister.
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 11:08 AM, Jon Ribbens wrote: > On 2016-02-22, Steven D'Aprano wrote: >> On Tue, 23 Feb 2016 05:48 am, Marko Rauhamaa wrote: >>> Jon Ribbens : I was under the impression that the point of UUIDs is that you can be *so* confident that there won't be a collision that for all practical purposes it's indistinguishable from being certain. >>> >>> Yes, if you generate a random 128-bit number, it will be unique -- >> >> If you generate a second random 128 bit number, you have a chance of 1 in >> 2**128 of a collision. All you can say is that it will be *very probably* >> unique. (I might even allow "almost certainly" unique.) > > If you are not prepared to say that something with a > 340282366920938463463374607431768211455 / > 340282366920938463463374607431768211456 chance of being true > is not "certainly true" then I'm not sure how you would not > be too scared to ever leave the house. Or not leave the house. > I mean, you're probably going to be hit by 10^25 meteorites, > which sounds painful. > >> If you generate 2**128 + 1 such numbers, you are *guaranteed* to > > ... have expired due to the heat death of the universe. Maybe... but by the time you get to 2**64 of them, you have a 50% chance of a collision. (That's either utterly intuitive or completely counter-intuitive, depending on who you are.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-22, Steven D'Aprano wrote: > On Tue, 23 Feb 2016 05:48 am, Marko Rauhamaa wrote: >> Jon Ribbens : >>> I was under the impression that the point of UUIDs is that you can be >>> *so* confident that there won't be a collision that for all practical >>> purposes it's indistinguishable from being certain. >> >> Yes, if you generate a random 128-bit number, it will be unique -- > > If you generate a second random 128 bit number, you have a chance of 1 in > 2**128 of a collision. All you can say is that it will be *very probably* > unique. (I might even allow "almost certainly" unique.) If you are not prepared to say that something with a 340282366920938463463374607431768211455 / 340282366920938463463374607431768211456 chance of being true is not "certainly true" then I'm not sure how you would not be too scared to ever leave the house. Or not leave the house. I mean, you're probably going to be hit by 10^25 meteorites, which sounds painful. > If you generate 2**128 + 1 such numbers, you are *guaranteed* to ... have expired due to the heat death of the universe. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-23, Steven D'Aprano wrote:
> On Tue, 23 Feb 2016 06:22 am, Jon Ribbens wrote:
>> Suppose you had code like this:
>>
>> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>>
>> Do we really think that is insecure or that there are any practical
>> attacks against it? It would be basically the same as saying that
>> urandom() is broken, surely?
>
> Correct. Any attack against urandom would be an attack on this. You would
> just have to trust that the kernel devs have made urandom as secure as
> possible, and pay no attention to what the man page says, as its wrong.
>
> By the way, Python 3.6 will have (once Guido formally approves it) a new
> module, "secrets", for securely generating (pseudo)random tokens like this:
>
> import secrets
> filename = secrets.token_hex(16)
+1
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, 23 Feb 2016 06:22 am, Jon Ribbens wrote:
> Suppose you had code like this:
>
> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>
> Do we really think that is insecure or that there are any practical
> attacks against it? It would be basically the same as saying that
> urandom() is broken, surely?
Correct. Any attack against urandom would be an attack on this. You would
just have to trust that the kernel devs have made urandom as secure as
possible, and pay no attention to what the man page says, as its wrong.
By the way, Python 3.6 will have (once Guido formally approves it) a new
module, "secrets", for securely generating (pseudo)random tokens like this:
import secrets
filename = secrets.token_hex(16)
https://www.python.org/dev/peps/pep-0506/
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 22 Feb 2016 22:50, "Ben Finney" wrote: > > Ethan Furman writes: > > > On 02/14/2016 04:08 PM, Ben Finney wrote: > > > > > I am unconcerned with whether there is a real filesystem entry of that > > > name; the goal entails having no filesystem activity for this. I want a > > > valid unique filesystem path, without touching the filesystem. > > > > This is impossible. If you don't touch the file system you have no > > way to know if the path is unique. > > That was unclear. Later in the same thread, I clarified that by “unique” > I mean nothing about entries already on the filesystem. Instead it means > “unpredictably different each time the function is called”. What does unpredictable mean in this context? Maybe I'm reading too much into that... What's wrong with the example I posted before? -- Oscar -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, 23 Feb 2016 06:22 am, Paul Rubin wrote:
> Chris Angelico writes:
>>> I was under the impression that the point of UUIDs is that you can be
>>> *so* confident that there won't be a collision that for all practical
>>> purposes it's indistinguishable from being certain.
>> Maybe, if everyone's cooperating. I'm not sure how they fare in the
>> face of malice though.
>
> There are different UUID algorithms, some of which have useful syntax
> but are easy to spoof. Uuid4 is random and implemented properly, should
> be hard to spoof.
I'm not sure what you mean by "spoof" in this context. Do you mean generate
collisions?
Do you mean "pretend to generate a UUID, but without actually doing so"?
That's how I interpret "spoof", but I don't quite understand why that would
be difficult. Here's one I just made now:
{00010203-0405-0607-0809-0a0b0c0d0e0f}
And another:
{836313e2-3b8a-53f2-9b90-0c9ade199e5d}
They weren't hard to spoof :-)
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, 23 Feb 2016 05:48 am, Marko Rauhamaa wrote: > Jon Ribbens : > >> I was under the impression that the point of UUIDs is that you can be >> *so* confident that there won't be a collision that for all practical >> purposes it's indistinguishable from being certain. > > Yes, if you generate a random 128-bit number, it will be unique -- If you generate a second random 128 bit number, you have a chance of 1 in 2**128 of a collision. All you can say is that it will be *very probably* unique. (I might even allow "almost certainly" unique.) If you generate 2**128 + 1 such numbers, you are *guaranteed* to have at least one collision. If I can arrange matters so that I am using the same seed as you, then I can generate the same UUIDs as you. If I know you are using the Mersenne Twister PRNG, and I can get hold of (by memory) 128 consecutive UUIDs, I can reconstruct the seed you are using and generate all future (and past) UUIDs the same as yours. (Well, when I say "I can", I don't mean *me*, I mean some attacker who is smarter than me, but not that much smarter.) > unless someone clones it. > > Cloning will be a practical issue when you clone virtual machines, for > example. This is certainly a practical issue that people have to be aware of. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Marko Rauhamaa writes: http://www.2uo.de/myths-about-urandom/ >> I don't know what web pamphlet you mean, > The only one linked above. Oh, I wouldn't have called that a pamphlet. I could quibble with the writing style but the points in the article are basically correct. > getrandom(2) is a good interface that distinguishes between the flag > values >0=> /dev/urandom >GRND_RANDOM => /dev/random >GRND_RANDOM | GRND_NONBLOCK => /dev/random (O_NONBLOCK) > However, although os.urandom() delegates to getrandom(), the > documentation suggests it uses the flag value 0 (/dev/urandom). Flag value 0 does the right thing and blocks if the entropy pool is not yet initialized, and doesn't block after that. That fixes the errors of both urandom (fails to block before there's enough entropy) and random (blocks even after there's enough entropy). The getrandom doc is also misleading about the workings of the entropy pools but that's ok. The actual algorithm is described here: http://www.pinkas.net/PAPERS/gpr06.pdf It's pretty clumsy but discussions about replacing it have gotten bogged down several times. OTOH maybe I'm out of date on this. >> The random/urandom interface was poorly designed and misleadingly >> documented. > It could be better I suppose, but I never found it particularly bad. The > nice thing about it is that it is readily usable in shell scripts. DJB describes the problems: https://groups.google.com/forum/#!msg/randomness-generation/4opmDHA6_3w/__TyKhbnNWsJ Regarding shell scripts, it should be a simple matter to put a wrapper around the system call. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 22Feb2016 12:34, Alan Bawden wrote: Cameron Simpson writes: On 16Feb2016 19:24, Alan Bawden wrote: So in the FIFO case, I might write something like the following: def make_temp_fifo(mode=0o600): while True: path = tempfile.mktemp() try: os.mkfifo(path, mode=mode) except FileExistsError: pass else: return path So is there something wrong with the above code? Other than the fact that the documentation says something scary about mktemp()? Well, it has a few shortcomings. It relies on mkfifo reliably failing if the name exists. It shounds like mkfifo is reliable this way, but I can imagine analogous use cases without such a convenient core action, and your code only avoids mktemp's security issue _because_ mkfifo has that fortuitous aspect. I don't understand your use of the word "fortuitous" here. mkfifo is defined to act that way according to POSIX. I wrote the code that way precisely because of that property. I sometimes write code knowing that adding two even numbers together results in an even answer. I suppose you might describe that as "fortuitous", but it's just things behaving as they are defined to behave! I mean here that your scheme isn't adaptable to a system call which will reuse an existing name. Of course, mkfifo, mkdir and open(.., O_EXCL) all have this nice feature. Secondly, why is your example better than:: os.mkfifo(os.path.join(mkdtemp(), 'myfifo')) My way is not much better, but I think it is a little better because your way I have to worry about deleting both the file and the directory when I am done, and I have to get the permissions right on two filesystem objects. (If I can use a TemporaryDirectory() context manager, the cleaning up part does get easier.) And it also seems wasteful to me, given that the way mkdtemp() is implemented is to generate a possible name, try creating it, and loop if the mkdir() call fails. (POSIX makes the same guarantee for mkdir() as it does for mkfifo().) Why not just let me do an equivalent loop myself? Go ahead. But I think Ben's specificly trying to avoid writing his own loop. On that basis, this example doesn't present a use case what can't be addressed by mkstemp or mkdtemp. Yes, if mktemp() were taken away from me, I could work around it. I'm just saying that in order to justify taking something like this away, it has to be both below some threshold of utility and above some threshold of dangerousness. In the canonical case of gets() in C, not only is fgets() almost a perfectly exact replacement for gets(), gets() is insanely dangerous. But the case of mktemp() doesn't seem to me to come close to this combination of redundancy and danger. You _do_ understand the security issue, yes? I sure looked like you did, until here. Well, it's always dangerous to say that you understand all the security issues of anything. In part that is why I wrote the code quoted above. I am open to the possibility that there is a security problem here that I haven't thought of. But so far the only problem anybody has with it is that you think there is something "fortuitous" about the way that it works. (As if that would be of any use in the situation above!) It looks like anxiety that some people might use mktemp() in a stupid way has caused an over-reaction. No, it is anxiety that mktemp's _normal_ use is inherently unsafe. So are you saying that the way I used mktemp() above is _abnormal_? In that you're not making a file. I mean "abnormal" in a statistical sense, and also in the "anticipated use case for mktemp's design". I'm not suggestioning you're wrong to use it like this. [ Here I have removed some perfectly reasonable text describing the race condition in question -- yes I really do understand that. ] This is neither weird nor even unlikely which is why kmtemp is strongly discouraged - naive (and standard) use is not safe. That you have contrived a use case where you can _carefully_ use mktemp in safety in no way makes mktemp recommendable. OK, so you _do_ seem to be saying that I have used mktemp() in a "contrived" and "non-standard" (and "non-naive"!) way. I'm genuinely surprised. I though I was just writing straightforward correct code and demonstrating that this was a useful utility that it was not hard to use safely. You seem to think what I did is something that ordinary programmers can not be expected to do. Your judgement is definitely different from mine! No, I meant only that (a) mktemp is normally used for regular files and (b) that mkdtemp()/mkfifo() present equivalent results without hand making a pick-a-name loop. Of course any programmer should be able to read the mktemp() spec and built from it. And ultimately this does all boil down to making judgements. It does make sense to remove things from libraries that are safety hazards (like gets() in C), I'm just trying to
Re: Make a unique filesystem path, without creating the file
On 02/22/2016 02:25 PM, Cameron Simpson wrote: On 22Feb2016 10:11, Ethan Furman wrote: On 02/14/2016 04:08 PM, Ben Finney wrote: I am unconcerned with whether there is a real filesystem entry of that name; the goal entails having no filesystem activity for this. I want a valid unique filesystem path, without touching the filesystem. This is impossible. If you don't touch the file system you have no way to know if the path is unique. I think Ben wants to avoid filesystem modification (let us ignore atime here). So one can read the filesystem to see what is current, but he does not want to actually make any new filesystem entry. Hmm -- well, he says "the goal entails having no filesystem activity for this", and seeing what already exists definitely requires file system activity . . . -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Ethan Furman writes: > On 02/14/2016 04:08 PM, Ben Finney wrote: > > > I am unconcerned with whether there is a real filesystem entry of that > > name; the goal entails having no filesystem activity for this. I want a > > valid unique filesystem path, without touching the filesystem. > > This is impossible. If you don't touch the file system you have no > way to know if the path is unique. That was unclear. Later in the same thread, I clarified that by “unique” I mean nothing about entries already on the filesystem. Instead it means “unpredictably different each time the function is called”. -- \ “It is difficult to get a man to understand something when his | `\ salary depends upon his not understanding it.” —Upton Sinclair, | _o__) 1935 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 22Feb2016 10:11, Ethan Furman wrote: On 02/14/2016 04:08 PM, Ben Finney wrote: I am unconcerned with whether there is a real filesystem entry of that name; the goal entails having no filesystem activity for this. I want a valid unique filesystem path, without touching the filesystem. This is impossible. If you don't touch the file system you have no way to know if the path is unique. I think Ben wants to avoid filesystem modification (let us ignore atime here). So one can read the filesystem to see what is current, but he does not want to actually make any new filesystem entry. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: How the heck does async/await work in Python 3.5
On 20.02.2016 07:53, Christian Gollwitzer wrote: If you have difficulties wit hthe overall concept, and if you are open to discussions in another language, take a look at this video: https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-39-await-co-routines MS has added coroutine support with very similar syntax to VC++ recently, and the developer tries to explain it to the "stackful" programmers. Thanks, Christian. Very informative video. Is something like shown in 12:50 ( cout << tcp_reader(1000).get() ) possible with asyncio? (tcp_reader would be async def) Best, Sven -- https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
BartC wrote: Our system must have been more advanced then, or designed for training. We used a time-sharing 'dec-system 10' and it was usually accessed via interactive terminals, either teletypes or the odd VDU. According to Wikipedia the first interactive version of Dartmouth BASIC appeared in 1964: https://en.wikipedia.org/wiki/Dartmouth_BASIC Also, the *very* earliest computer systems were all interactive -- you sat in front of a panel flipping switches and reading lights! -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Paul Rubin : >>> http://www.2uo.de/myths-about-urandom/ >> Did you post the link because you agreed with the Web pamphlet? > > I don't know what web pamphlet you mean, The only one linked above. Cryptography is tricky business, indeed. I know enough about it not to improvise too much. Infinitesimal weaknesses can make a difference between feasible and unfeasible attacks. > but the right thing to use now is getrandom(2). getrandom(2) is a good interface that distinguishes between the flag values 0=> /dev/urandom GRND_RANDOM => /dev/random GRND_RANDOM | GRND_NONBLOCK => /dev/random (O_NONBLOCK) However, although os.urandom() delegates to getrandom(), the documentation suggests it uses the flag value 0 (/dev/urandom). > The random/urandom interface was poorly designed and misleadingly > documented. It could be better I suppose, but I never found it particularly bad. The nice thing about it is that it is readily usable in shell scripts. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Marko Rauhamaa writes: >> http://www.2uo.de/myths-about-urandom/ > Did you post the link because you agreed with the Web pamphlet? I don't know what web pamphlet you mean, but the right thing to use now is getrandom(2). The random/urandom interface was poorly designed and misleadingly documented. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Random832 : > On Mon, Feb 22, 2016, at 14:32, Marko Rauhamaa wrote: >> urandom() is not quite random and so should not be considered >> cryptographically airtight. >> >> Under Linux, /dev/random is the way to go when strong security is >> needed. Note that /dev/random is a scarce resource on ordinary >> systems. > > http://www.2uo.de/myths-about-urandom/ Did you post the link because you agreed with the Web pamphlet? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Mon, Feb 22, 2016, at 14:32, Marko Rauhamaa wrote: > urandom() is not quite random and so should not be considered > cryptographically airtight. > > Under Linux, /dev/random is the way to go when strong security is > needed. Note that /dev/random is a scarce resource on ordinary systems. http://www.2uo.de/myths-about-urandom/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 6:22 AM, Jon Ribbens
wrote:
>> Maybe, if everyone's cooperating. I'm not sure how they fare in the
>> face of malice though.
>
> Suppose you had code like this:
>
> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>
> Do we really think that is insecure or that there are any practical
> attacks against it? It would be basically the same as saying that
> urandom() is broken, surely?
Sure, that would be safe. But UUIDs aren't necessarily based on "give
me sixteen bytes from urandom". They can involve
potentially-predictable information such as MAC addresses, current
time of day, and so on, which gives them significantly less
randomness. In that kind of usage, they're not intended to be
cryptographically secure.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Jon Ribbens :
> Suppose you had code like this:
>
> filename = binascii.hexlify(os.urandom(16)).decode("ascii")
>
> Do we really think that is insecure or that there are any practical
> attacks against it? It would be basically the same as saying that
> urandom() is broken, surely?
urandom() is not quite random and so should not be considered
cryptographically airtight.
Under Linux, /dev/random is the way to go when strong security is
needed. Note that /dev/random is a scarce resource on ordinary systems.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-22, Chris Angelico wrote:
> On Tue, Feb 23, 2016 at 5:39 AM, Jon Ribbens
> wrote:
>> On 2016-02-22, Chris Angelico wrote:
>>> On Tue, Feb 23, 2016 at 5:17 AM, Jon Ribbens
>>> wrote:
Weell, I have a lot of sympathy for that point, but on the other
hand the whole concept of UUIDs ("import uuid") is predicated on the
opposite assumption.
>>>
>>> Not quite opposite. Ethan is asserting that you cannot be *certain*
>>> without actually checking the FS; the point of UUIDs is that you can
>>> be fairly *confident* that there won't be a collision. There is a
>>> nonzero probability of accidental collisions, and if an attacker is
>>> deliberately trying to _force_ a collision, it's most definitely
>>> possible. So both views are correct.
>>
>> I was under the impression that the point of UUIDs is that you can be
>> *so* confident that there won't be a collision that for all practical
>> purposes it's indistinguishable from being certain.
>
> Maybe, if everyone's cooperating. I'm not sure how they fare in the
> face of malice though.
Suppose you had code like this:
filename = binascii.hexlify(os.urandom(16)).decode("ascii")
Do we really think that is insecure or that there are any practical
attacks against it? It would be basically the same as saying that
urandom() is broken, surely?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Chris Angelico writes: >> I was under the impression that the point of UUIDs is that you can be >> *so* confident that there won't be a collision that for all practical >> purposes it's indistinguishable from being certain. > Maybe, if everyone's cooperating. I'm not sure how they fare in the > face of malice though. There are different UUID algorithms, some of which have useful syntax but are easy to spoof. Uuid4 is random and implemented properly, should be hard to spoof. -- https://mail.python.org/mailman/listinfo/python-list
Re: 0x80070570-The file or directory is corrupted and unreadable
El martes, 22 de diciembre de 2015, 13:46:01 (UTC-3), eryk sun escribió: > On Tue, Dec 22, 2015 at 8:02 AM, muizz hasan wrote: > > Hi there! I've been recently trying to install Python for Windows 10 > > and I've been encountering some issues. Every time i try to install > > the program it just says"0x80070570-The file or directory is corrupted > > and unreadable". I have attached my log file and i hope that you guys > > might enlighten me on how to solve my problem. Thank you! > > Try downloading a new copy of the installer. Clear your browser cache first. Thank you!! this was usefull!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 5:39 AM, Jon Ribbens
wrote:
> On 2016-02-22, Chris Angelico wrote:
>> On Tue, Feb 23, 2016 at 5:17 AM, Jon Ribbens
>> wrote:
>>> Weell, I have a lot of sympathy for that point, but on the other
>>> hand the whole concept of UUIDs ("import uuid") is predicated on the
>>> opposite assumption.
>>
>> Not quite opposite. Ethan is asserting that you cannot be *certain*
>> without actually checking the FS; the point of UUIDs is that you can
>> be fairly *confident* that there won't be a collision. There is a
>> nonzero probability of accidental collisions, and if an attacker is
>> deliberately trying to _force_ a collision, it's most definitely
>> possible. So both views are correct.
>
> I was under the impression that the point of UUIDs is that you can be
> *so* confident that there won't be a collision that for all practical
> purposes it's indistinguishable from being certain.
Maybe, if everyone's cooperating. I'm not sure how they fare in the
face of malice though.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: PyPDF2 merge / out of memory
Le dimanche 21 février 2016 11:42:33 UTC+1, cpol...@gmail.com a écrit : > Hello, > > There is an issue with PyPDF2 and merging file > https://github.com/mstamy2/PyPDF2/issues/189 > > Does anybody know an alternate library to merge PDF and produce optimized pdf > file ? > > Thanks a lot > > Clement Thanks Mark, I'll dig these tools. Regards, Clement -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Jon Ribbens : > I was under the impression that the point of UUIDs is that you can be > *so* confident that there won't be a collision that for all practical > purposes it's indistinguishable from being certain. Yes, if you generate a random 128-bit number, it will be unique -- unless someone clones it. Cloning will be a practical issue when you clone virtual machines, for example. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-22, Chris Angelico wrote:
> On Tue, Feb 23, 2016 at 5:17 AM, Jon Ribbens
> wrote:
>> Weell, I have a lot of sympathy for that point, but on the other
>> hand the whole concept of UUIDs ("import uuid") is predicated on the
>> opposite assumption.
>
> Not quite opposite. Ethan is asserting that you cannot be *certain*
> without actually checking the FS; the point of UUIDs is that you can
> be fairly *confident* that there won't be a collision. There is a
> nonzero probability of accidental collisions, and if an attacker is
> deliberately trying to _force_ a collision, it's most definitely
> possible. So both views are correct.
I was under the impression that the point of UUIDs is that you can be
*so* confident that there won't be a collision that for all practical
purposes it's indistinguishable from being certain.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 2016-02-22, Ethan Furman wrote:
> On 02/14/2016 04:08 PM, Ben Finney wrote:
>> I am unconcerned with whether there is a real filesystem entry of that
>> name; the goal entails having no filesystem activity for this. I want a
>> valid unique filesystem path, without touching the filesystem.
>
> This is impossible. If you don't touch the file system you have no way
> to know if the path is unique.
Weell, I have a lot of sympathy for that point, but on the other
hand the whole concept of UUIDs ("import uuid") is predicated on the
opposite assumption.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On Tue, Feb 23, 2016 at 5:17 AM, Jon Ribbens
wrote:
> On 2016-02-22, Ethan Furman wrote:
>> On 02/14/2016 04:08 PM, Ben Finney wrote:
>>> I am unconcerned with whether there is a real filesystem entry of that
>>> name; the goal entails having no filesystem activity for this. I want a
>>> valid unique filesystem path, without touching the filesystem.
>>
>> This is impossible. If you don't touch the file system you have no way
>> to know if the path is unique.
>
> Weell, I have a lot of sympathy for that point, but on the other
> hand the whole concept of UUIDs ("import uuid") is predicated on the
> opposite assumption.
Not quite opposite. Ethan is asserting that you cannot be *certain*
without actually checking the FS; the point of UUIDs is that you can
be fairly *confident* that there won't be a collision. There is a
nonzero probability of accidental collisions, and if an attacker is
deliberately trying to _force_ a collision, it's most definitely
possible. So both views are correct.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
On 02/14/2016 04:08 PM, Ben Finney wrote: I am unconcerned with whether there is a real filesystem entry of that name; the goal entails having no filesystem activity for this. I want a valid unique filesystem path, without touching the filesystem. This is impossible. If you don't touch the file system you have no way to know if the path is unique. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: Make a unique filesystem path, without creating the file
Cameron Simpson writes: > On 16Feb2016 19:24, Alan Bawden wrote: >>So in the FIFO case, I might write something like the following: >> >>def make_temp_fifo(mode=0o600): >>while True: >>path = tempfile.mktemp() >>try: >>os.mkfifo(path, mode=mode) >>except FileExistsError: >>pass >>else: >>return path >> >>So is there something wrong with the above code? Other than the fact >>that the documentation says something scary about mktemp()? > > Well, it has a few shortcomings. > > It relies on mkfifo reliably failing if the name exists. It shounds like > mkfifo is reliable this way, but I can imagine analogous use cases without > such a convenient core action, and your code only avoids mktemp's security > issue _because_ mkfifo has that fortuitous aspect. I don't understand your use of the word "fortuitous" here. mkfifo is defined to act that way according to POSIX. I wrote the code that way precisely because of that property. I sometimes write code knowing that adding two even numbers together results in an even answer. I suppose you might describe that as "fortuitous", but it's just things behaving as they are defined to behave! > Secondly, why is your example better than:: > > os.mkfifo(os.path.join(mkdtemp(), 'myfifo')) My way is not much better, but I think it is a little better because your way I have to worry about deleting both the file and the directory when I am done, and I have to get the permissions right on two filesystem objects. (If I can use a TemporaryDirectory() context manager, the cleaning up part does get easier.) And it also seems wasteful to me, given that the way mkdtemp() is implemented is to generate a possible name, try creating it, and loop if the mkdir() call fails. (POSIX makes the same guarantee for mkdir() as it does for mkfifo().) Why not just let me do an equivalent loop myself? > On that basis, this example doesn't present a use case what can't be > addressed by mkstemp or mkdtemp. Yes, if mktemp() were taken away from me, I could work around it. I'm just saying that in order to justify taking something like this away, it has to be both below some threshold of utility and above some threshold of dangerousness. In the canonical case of gets() in C, not only is fgets() almost a perfectly exact replacement for gets(), gets() is insanely dangerous. But the case of mktemp() doesn't seem to me to come close to this combination of redundancy and danger. > You _do_ understand the security issue, yes? I sure looked like you did, > until here. Well, it's always dangerous to say that you understand all the security issues of anything. In part that is why I wrote the code quoted above. I am open to the possibility that there is a security problem here that I haven't thought of. But so far the only problem anybody has with it is that you think there is something "fortuitous" about the way that it works. >>(As if that would be of any use in the >>situation above!) It looks like anxiety that some people might use >>mktemp() in a stupid way has caused an over-reaction. > > No, it is anxiety that mktemp's _normal_ use is inherently unsafe. So are you saying that the way I used mktemp() above is _abnormal_? > [ Here I have removed some perfectly reasonable text describing the > race condition in question -- yes I really do understand that. ] > > This is neither weird nor even unlikely which is why kmtemp is strongly > discouraged - naive (and standard) use is not safe. > > That you have contrived a use case where you can _carefully_ use mktemp in > safety in no way makes mktemp recommendable. OK, so you _do_ seem to be saying that I have used mktemp() in a "contrived" and "non-standard" (and "non-naive"!) way. I'm genuinely surprised. I though I was just writing straightforward correct code and demonstrating that this was a useful utility that it was not hard to use safely. You seem to think what I did is something that ordinary programmers can not be expected to do. Your judgement is definitely different from mine! And ultimately this does all boil down to making judgements. It does make sense to remove things from libraries that are safety hazards (like gets() in C), I'm just trying to argue that mktemp() isn't nearly dangerous enough to deserve more than a warning in its documentation. You don't agree. Oh well... Up until this point, you haven't said anything that I actually think is flat out wrong, we just disagree about what tools it is reasonable to take away from _all_ programmers just because _some_ programmers might use them to make a mess. > In fact your use case isn't safe, because _another_ task using mktemp > in conflict as a plain old temporary file may grab your fifo. But here in very last sentence I really must disagree. If the code I wrote above is "unsafe" because some _other_ process might be using mktemp() badly and stumble over
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On 22/02/2016 10:46, Steven D'Aprano wrote: On Mon, 22 Feb 2016 08:52 am, Jussi Piitulainen wrote: BartC writes: IIRC, the first programming exercise I ever did (in 1976 using Algol 60) involved reading 3 numbers from the teletype and working out if those could form sides of a triangle. That was a lousy user interface even then - an inflexible user interaction without even a possibility of handling errors interactively? Command line arguments would have been better (if available, that is). Jussi, I think you have an inflated expectation of what was available in 1976. Command line? What's that? Programs ran in batch mode, and 'interactive' meant that you could easily slip out one punched card, replace it with a different one, and run the program again. Our system must have been more advanced then, or designed for training. We used a time-sharing 'dec-system 10' and it was usually accessed via interactive terminals, either teletypes or the odd VDU. It still supported punched cards but that was more because they were still being used in the real world. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: avoid for loop calling Generator function
On Mon, Feb 22, 2016 at 8:38 AM, Arshpreet Singh wrote:
> On Monday, 22 February 2016 19:05:24 UTC+5:30, Peter Otten wrote:
>> or the slightly less convoluted
>>
>> sys.stdout.writelines(map("{}\n".format, read_pdf("book.pdf")))
>
> Actually I am using this function in Android App which is being built
using Kivy, Where I am returning whole text into a file, So what you think
will be more efficient way?
Profile them and find out, but I don't think you'll find the difference is
great enough to be overly concerned with. Pick the way that is more
readable and doesn't introduce any gross inefficiencies (such as
concatenating strings in a loop).
> But when I am calling pdf_read() from nother function to avoid for loop
why it is not working?
> say:
>
> def hello()
> yield from read_pdf('book.pdf')
This uses yield from, which makes it a generator function.
>
> print(hello()) # still returns memory location instead of text. If I am
not wrong yield from can be used to avoid for loop?
hello is a generator function, so calling it just creates a generator
object. Printing it then prints out the repr of that generator object,
which is just something like .
Notably, you haven't actually *executed* the generator object, which would
require iterating over it, e.g.:
for i in hello():
print(i)
So you haven't actually avoided creating a for loop; you've just added a
redundant layer between the for loop and the thing it's actually iterating
over.
--
https://mail.python.org/mailman/listinfo/python-list
Re: avoid for loop calling Generator function
On Tue, Feb 23, 2016 at 2:38 AM, Arshpreet Singh wrote:
>> next(filter(print, read_pdf("book.pdf")), None)
>
> Why we are w=using filter here?
It's a beautiful hack. It'll filter according to the "print"
predicate, which always returns None, and will thus filter everything
out. One single call to next() will thus process and print the entire
PDF, and then - since it has a second parameter - return None instead
of raising StopIteration.
Or maybe it's a gross and ugly hack, with the exact same description.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: avoid for loop calling Generator function
On Monday, 22 February 2016 19:05:24 UTC+5:30, Peter Otten wrote:
> Arshpreet Singh wrote:
>
> > Hi, I am converting PDF into text file, I am using following code.
> >
> > from pypdf2 import PdfFileReader
> >
> > def read_pdf(pdfFileName):
> >
> > pdf = PdfFileReader(pdfFileName)
> >
> > yield from (pg.extractText() for pg in pdf.pages)
> >
> > for i in read_pdf('book.pdf'):
> > print(i)
> >
> > I want to avoid for loop , I also tried to create another function and
> > call read_pdf() inside that new function using yield from but I think I am
> > missing real picture here
>
> While it is possible to replace the loop with
>
> next(filter(print, read_pdf("book.pdf")), None)
Why we are w=using filter here?
> or the slightly less convoluted
>
> sys.stdout.writelines(map("{}\n".format, read_pdf("book.pdf")))
Actually I am using this function in Android App which is being built using
Kivy, Where I am returning whole text into a file, So what you think will be
more efficient way?
> the for loop is the obvious and therefore recommended solution. Personally,
> I would also replace
>
> > yield from (pg.extractText() for pg in pdf.pages)
>
> with the good old
>
> for pg in pdf.pages:
> yield pg.extractText()
>
> and reserve the generator expression for occasions where it has a
> demonstrable advantage in readability.
But when I am calling pdf_read() from nother function to avoid for loop why it
is not working?
say:
def hello()
yield from read_pdf('book.pdf')
print(hello()) # still returns memory location instead of text. If I am not
wrong yield from can be used to avoid for loop?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On Mon, Feb 22, 2016, at 05:46, Steven D'Aprano wrote: > Jussi, I think you have an inflated expectation of what was available in > 1976. Command line? What's that? Programs ran in batch mode, > and 'interactive' meant that you could easily slip out one punched card, > replace it with a different one, and run the program again. User experience was hardly uniform across different systems in that era. The book "Hackers", for example, describes an interactive computer that was used at MIT in *1959*. More relevantly to the lineage of the systems we use today, PDP-7 Unix was first developed in 1969 - and PDP-11 6th Edition Unix, very close to a recognizable modern system, was released in 1975. -- https://mail.python.org/mailman/listinfo/python-list
Re: avoid for loop calling Generator function
Arshpreet Singh wrote:
> Hi, I am converting PDF into text file, I am using following code.
>
> from pypdf2 import PdfFileReader
>
> def read_pdf(pdfFileName):
>
> pdf = PdfFileReader(pdfFileName)
>
> yield from (pg.extractText() for pg in pdf.pages)
>
> for i in read_pdf('book.pdf'):
> print(i)
>
> I want to avoid for loop , I also tried to create another function and
> call read_pdf() inside that new function using yield from but I think I am
> missing real picture here
While it is possible to replace the loop with
next(filter(print, read_pdf("book.pdf")), None)
or the slightly less convoluted
sys.stdout.writelines(map("{}\n".format, read_pdf("book.pdf")))
the for loop is the obvious and therefore recommended solution. Personally,
I would also replace
> yield from (pg.extractText() for pg in pdf.pages)
with the good old
for pg in pdf.pages:
yield pg.extractText()
and reserve the generator expression for occasions where it has a
demonstrable advantage in readability.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On Mon, Feb 22, 2016 at 11:51 PM, BartC wrote:
> Yes, I mentioned that point. In the OP's language, the variables can be
> declared as a particular type; in Python they could perhaps be 'declared' by
> first assigning with 0, 0.0 or "" for example. But that would need reference
> parameters to make use of tidily.
What you could do is something like this:
def read_values(file, values):
"""Read one line from file and parse it into the given list.
Prepopulate values with a series of types or instances of types.
The line of text read will be parsed accordingly.
Returns the number of elements successfully read.
"""
line = file.readline()
for idx, val in enumerate(values):
if not isinstance(val, type):
val = type(val)
# Now add the logic to read an int, a float,
# a str, a Fraction, etc etc etc, from the line,
# breaking out of the loop if one can't be
# read, and stashing the result into values[idx]
# if it can.
return idx
with open("inputfile") as f:
values = [int, int, float, Fraction, str]
while read_values(f, values) == 5:
[index, level, percent, ratio, message] = values
This lets you "declare" the values' types, and use an input/output
parameter. It's not exactly the most Pythonic of techniques, but it
does kinda work. I suspect, though, that what you'd _really_ want is
something more like a sscanf string, or a "typed regex", which would
allow you to specify a bit more flexibly what you're looking for. You
could easily make a template string that has magic markers for "take
an integer", which would insert [0-9]+ (or something more elaborate)
and then capture it and call int() on the string before returning it.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On 22/02/2016 11:16, Steven D'Aprano wrote: On Mon, 22 Feb 2016 12:16 am, BartC wrote: [...] No need for anyone to re-invent the wheel! ;-) I keep seeing this in the thread. Python has all this capability, yet it still requires a lot of fiddly code to be added to get anywhere near as simple as this: read f, a, b, c I can't say I really know what that means. My guess is that reads three things (a, b and c) from a file f, but what those things are (strings, binary blobs, floats, ints, something else?) is a mystery. Yes, I mentioned that point. In the OP's language, the variables can be declared as a particular type; in Python they could perhaps be 'declared' by first assigning with 0, 0.0 or "" for example. But that would need reference parameters to make use of tidily. But even when declared, sometimes additional formatting info is needed (numbers might be in hex or binary for example). One of the major problems with output parameters is that it isn't obvious what is an output parameter and what isn't. When it comes to read, perhaps you can guess, but when it comes to arbitrary functions, who can tell what is output and what is input? fromaginate m, x, y, z, a, b Whereas a functional design makes it obvious: output is on the left of the assignment symbol, input is on the right. x, y, z = fromaginate m, a, b Old-style /statements/ such as 'read' aren't functional. Then it is easy to specify that the parameters need to be l-values. readline(f, a, b, c) I can't see a straightforward way of making this possible while still keeping a, b and c simple integer, float or string types (because Python's reference parameters don't work quite the right way). Python doesn't have reference parameters. Please feel free to discuss further if you disagree, or want additional explanation. Well, I've just finished reimplementing a language so that it uses CPython-like references for objects (not, however, for types such as small integers which stay as value types). But that language retains the use of pointers which can be used /as well as/ references to provide pass-by-reference, even for integers. Then it is possible to write a function which can be called like: readline(f,a,b,c) and which would update the caller's a,b,c variables. Python can't do this, even with 100% reference objects. This is because most types are immutable and their values can be shared across unrelated variables. Try and change one of those, and all would be changed! In any case, doing an assignment to a parameter just replaces its local value. Caller's data can only be changed via an in-place modification of the parameter, which I believe only works for lists. (Maybe, something like this can be done with classes, but I did say you can't do it with simple types.) -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
EuroPython 2016: Early bird ticket sales
After the Call for Proposals for EuroPython 2016 last week, we are now calling out to everyone interested in signing up as EuroPython attendee early. You will be able to benefit from reduced ticket prices for a short period of time. Our early bird ticket sales are limited to 300 tickets. Regular sales start shortly after we’ve ended the early bird sales. *** https://ep2016.europython.eu/en/registration/ *** More Information and Ticket Shop We will be opening the flood gates on Tuesday, 09:00 CET, so set your alarms to make sure you can secure your ticket. Experience shows that early bird tickets sell out quickly. PS: Remember to submit your proposals for the conference. There are less than two weeks left: https://ep2016.europython.eu/en/call-for-proposals/ With gravitational regards, -- EuroPython 2016 Team http://ep2016.europython.eu/ http://www.europython-society.org/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On 22/02/2016 08:50, Jussi Piitulainen wrote:
BartC writes:
Reading stuff from an interactive console or terminal, is such an
important part of the history of computing, still being used now, that
you'd think a language would provide simple, ready-to-use methods ways
to do it.
I think it does. Do you at least agree that it provides a reasonably
simple way to ask for a single line of text?
stuff = input()
Is that already too obscure or advanced for you? Because surely not?
That's a good starting point. It reads an entire line of text, as a
string. So that's part of the job done. The rest of it needs some string
processing.
But even 'input' has some issues. While it seemed to work, the docs I
looked at suggested it interpreted the input as an expression, and I
needed to use raw_input().
Was this a Python 2/3 problem? The docs I used
(https://en.wikibooks.org/wiki/Python_Programming/Input_and_Output)
didn't appear to mention that, not in the title or around the edges
anyway! Eventually I saw that as a italicised note within the text.
So, it's input() on 3.x and raw_input() on 2.x.
But another thing is that it doesn't scale up to read from a file, as
far as I can see. File reading is rather different, and it behaves
differently regarding line endings.
(For comparison, here's how it works on another language I use:
readln a,b,c
This waits for a line of input (it becomes the read buffer), then reads
three integers from the buffer.
read d,e
This reads two more integers from that buffer (if there is no more data,
it reads zeros).
readln @f,a,b,c
This does the same, from file handle f.
The equivalent of Python's input() might be:
s := sreadln()# from console
s := sreadln(f) # from a file
s := sreadln("10,20") # from a string
but the entire line is also placed into the read buffer so that
subsequent read commands (or sread() function calls) can read individual
values as before.
Reading anything other than integers is bit more fiddly (you can't have
everything!) So that means using something like this:
read a:"r", b:"h", c:"s", d:"n"
So reading as a float, an int using hexadecimal, a string (delimited by
white space or punctuation, or quoted), a name (file names etc), and so on.
There are limitations, but this stuff is there when I don't need
anything more sophisticated.)
--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Setting up/authenticating Google API?
On Sunday, 21 February 2016 21:21:54 UTC+5:30, Skip Montanaro wrote: > This isn't strictly a Python question, however... Once I get myself > authenticated, I intend to use the Python Google API to pump archived > mail messages from a few defunct mailing lists into Google Groups. I > thought it would be pretty straightforward, but my attempts a few > months ago were completely unsuccessful. I asked for help on the > relevant Google product forum but got zero assistance. > > I'd really appreciate a little help with the setup. If anyone has had > successful experience with this, please contact me off-list. Once I > get something going, I'll write a blog post so others don't have the > same problems I have. > > Thanks, > > Skip Montanaro Your question seems quite cinfusing to me but I think following may is what you are asking for.(Read the source) https://google-mail-oauth2-tools.googlecode.com/svn/trunk/python/oauth2.py After getting the authentication you can use imaplib to get all the gmail Data. -- https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
Steven D'Aprano writes: > On Mon, 22 Feb 2016 08:52 am, Jussi Piitulainen wrote: > >> BartC writes: > >>> IIRC, the first programming exercise I ever did (in 1976 using Algol >>> 60) involved reading 3 numbers from the teletype and working out if >>> those could form sides of a triangle. >> >> That was a lousy user interface even then - an inflexible user >> interaction without even a possibility of handling errors >> interactively? Command line arguments would have been better (if >> available, that is). > > Jussi, I think you have an inflated expectation of what was available > in 1976. Command line? What's that? Programs ran in batch mode, and > 'interactive' meant that you could easily slip out one punched card, > replace it with a different one, and run the program again. You are probably right. I was thinking ten years off. Sorry. But if that's how one worked a teletype, then it sounds similar to command line arguments to me. Or to a configuration file that the program then when run. Not the interaction-prompting interface that I was thinking of when I called it lousy. -- https://mail.python.org/mailman/listinfo/python-list
avoid for loop calling Generator function
Hi, I am converting PDF into text file, I am using following code.
from pypdf2 import PdfFileReader
def read_pdf(pdfFileName):
pdf = PdfFileReader(pdfFileName)
yield from (pg.extractText() for pg in pdf.pages)
for i in read_pdf('book.pdf'):
print(i)
I want to avoid for loop , I also tried to create another function and call
read_pdf() inside that new function using yield from but I think I am missing
real picture here
--
https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On Mon, 22 Feb 2016 12:16 am, BartC wrote: [...] >> No need for anyone to re-invent the >> wheel! ;-) > > I keep seeing this in the thread. Python has all this capability, yet it > still requires a lot of fiddly code to be added to get anywhere near as > simple as this: > >read f, a, b, c I can't say I really know what that means. My guess is that reads three things (a, b and c) from a file f, but what those things are (strings, binary blobs, floats, ints, something else?) is a mystery. One of the major problems with output parameters is that it isn't obvious what is an output parameter and what isn't. When it comes to read, perhaps you can guess, but when it comes to arbitrary functions, who can tell what is output and what is input? fromaginate m, x, y, z, a, b Whereas a functional design makes it obvious: output is on the left of the assignment symbol, input is on the right. x, y, z = fromaginate m, a, b > And this is code that is not going to be obvious to anyone starting out. > Even accepting that syntax limitations might require this to be written > as: > >readline(f, a, b, c) > > I can't see a straightforward way of making this possible while still > keeping a, b and c simple integer, float or string types (because > Python's reference parameters don't work quite the right way). Python doesn't have reference parameters. Please feel free to discuss further if you disagree, or want additional explanation. > (There is also the question of 'readline' knowing what types of values > to read. This information would not be needed in Fortran or Basic but > somehow needs to be supplied here, if a particular set of types is to > imposed on the input.) > > In other words, it seems this particular wheel does require re-inventing! Hmmm, well, yes, I think perhaps it is fair to say that there is middle ground where Python misses out. Python's raw IO is quite powerful, and it can handle unstructured binary of text files with no difficulty. It also comes with libraries for certain common file formats, like CSV, XML, JSON and others. But there's a middle ground, of *minimally structured* text files, where Python leaves it up to you. Admittedly it's not difficult, any minimally competent Python programmer should be able to read a bunch of ints or floats from a text file without working up a sweat, but beginners may find this tricky. So I think you are right: there is a narrow, but useful, niche of semi-structured textual data that BASIC and VB support that Python doesn't support out of the box. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
On Mon, 22 Feb 2016 08:52 am, Jussi Piitulainen wrote: > BartC writes: >> IIRC, the first programming exercise I ever did (in 1976 using Algol >> 60) involved reading 3 numbers from the teletype and working out if >> those could form sides of a triangle. > > That was a lousy user interface even then - an inflexible user > interaction without even a possibility of handling errors interactively? > Command line arguments would have been better (if available, that is). Jussi, I think you have an inflated expectation of what was available in 1976. Command line? What's that? Programs ran in batch mode, and 'interactive' meant that you could easily slip out one punched card, replace it with a different one, and run the program again. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: can try expect have if else.
On Mon, 22 Feb 2016 03:12 am, Ganesh Pal wrote:
> Hi Team,
>
> Iam on python 2.6 , need input on the below piece of code.
>
>
> EXIT_STATUS_ERROR=1
>
> def create_dataset():
> """
> """
> logging.info("Dataset create.Started !!!")
> try:
> if os.path.ismount("/nfs_mount"):
> touch_file("inode_fixcrc.txt")
> logging.info("Dataset create.Done !!!")
> else:
> raise Exception("/nfs_mount is not mounted. Dataset create
> failed !!!")
> return False
You should not raised Exception directly.
The "return False" line is dead code, it will never be executed.
> except Exception as e:
> logging.error(e)
> sys.stderr.write("Dataset create failed...Exiting !!!")
> sys.exit(EXIT_STATUS_ERROR)
> return True
>
> 1. Can we have if else with in a try except block
Did you try it to see what would happen?
Yes, you can have an if...else block inside a try block.
> 2. How can the above code be improved
# Use a custom exception type.
class CreateDatasetError(OSError):
pass
def touch_file(path, filename):
"""Raises OSError or IOError if path is not a mounted mount point,
and path/filename cannot be created.
"""
if not os.path.ismount(path):
raise CreateDatasetError('%s is not mounted, create failed.' % path)
else:
fullname = os.join(path, filename)
f = open(fullname, 'w')
f.close()
EXIT_STATUS_ERROR=1
def create_dataset():
"""
"""
logging.info("Dataset create.Started !!!")
try:
touch_file("/nfs_mount", "inode_fixcrc.txt")
except (IOError, OSError) as e:
logging.error(e)
sys.stderr.write("Dataset create failed...Exiting !!!")
sys.exit(EXIT_STATUS_ERROR)
logging.info("Dataset create.Done !!!")
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Scipy2016: call for proposals
Dear all, SciPy 2016, the Fifteenth Annual Conference on Python in Science, takes place in Austin, TX on July, 11th to 17th. The conference features two days of tutorials by followed by three days of presentations, and concludes with two days of developer sprints on projects of interest to attendees. . The topics presented at SciPy are very diverse, with a focus on advanced software engineering and original uses of Python and its scientific libraries, either in theoretical or experimental research, from both academia and the industry. This year we are happy to announce two specialized tracks that run in parallel to the general conference (Data Science , High Performance Computing) and 8 mini-symposia (Earth and Space Science, Biology and Medicine, Engineering, Social Sciences, Special Purpose Databases, Case Studies in Industry, Education, Reproducibility) Submissions for talks and posters are welcome on our website ( http://scipy2016.scipy.org). In your abstract, please provide details on what Python tools are being employed, and how. The talk and poster submission deadline is March 25th, 2016, while the tutorial submission deadline is March, 21st, 2016. Important dates: Mar 21: Tutorial Proposals Due Mar 25: Talk and Poster Proposals Due May 11: Plotting Contest Submissions Due Apr 22: Tutorials Announced Apr 22: Financial Aid Submissions Due May 4: Talk and Posters Announced May 11: Financial Aid Recipients Notified May 22: Early Bird Registration Deadline Jul 11-12: SciPy 2016 Tutorials Jul 13-15: SciPy 2016 General Conference Jul 16-17: SciPy 2016 Sprints We look forward to an exciting conference and hope to see you in Austin in July! The Scipy 2016 http://scipy2016.scipy.org/ Conference Chairs: Aric Hagberg, Prabhu Ramachandran Tutorial Chairs: Justin Vincent, Ben Root Program Chair: Serge Rey, Nelle Varoquaux Proceeding Chairs: Sebastian Benthall -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLite
On Mon, Feb 22, 2016 at 7:40 PM, Klaus Jantzen wrote: >That did not work because I did not install Python 3.5. with apt-get >but downloaded the source and compiled myself. >Thus apt-get does not have any information about the Python 3.5 >installation. It generally does work; the build dependencies for CPython 3.2 are the same as for CPython 3.5, with maybe one or two new ones added for new modules. So you can ask apt-get to grab you the deps for 3.2 and then build 3.5 from source. That's what I usually do when I build from source control - whether it's CPython or Pike or git or FFMpeg or anything else, I can generally depend on the build deps for the old version being good enough to build the new version. Even when they're not exactly right, they're close enough that it saves a lot of time. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
BartC writes:
> On 21/02/2016 21:52, Jussi Piitulainen wrote:
[...]
> def istriangle(*threeintegers):
>> a,b,c = sorted(threeintegers)
>> return a*a + b*b == c*c
>
> (That detects right-angled triangles. I think 'return a+b>c' is the
> correct test for any triangle. It becomes trivial when you have sort
> available, but that's part of the point of the exercise.)
Sorry about that. (Non-positive sides should also be ruled out.)
[...]
> Reading stuff from an interactive console or terminal, is such an
> important part of the history of computing, still being used now, that
> you'd think a language would provide simple, ready-to-use methods ways
> to do it.
I think it does. Do you at least agree that it provides a reasonably
simple way to ask for a single line of text?
stuff = input()
Is that already too obscure or advanced for you? Because surely not?
It can be used piecemeal:
print("a b c = ")
stuff = input()
parts = stuff.split()
a = int(parts[0])
b = int(parts[1])
c = int(parts[2])
The same can be done in one statement:
a, b, c = map(int, input("a b c = ").split())
Here's a simpler user interface:
a = int(input("a = "))
b = int(input("b = "))
c = int(input("c = "))
These building blocks can be used to define more complex dialogues in
obvious ways, probably using exception handling.
def please(*types, *, prompt = None):
"""Repeatedly prompts the console user for space-separated values in
one line until an input line matches the given types. Returns the
corresponding values. At end of input, raises UpsetException."""
...
a, b, c = please(int, int, int, prompt = "a b c = ")
def readWithDWIM(source, *types):
"""Reads and returns whitespace-separated values from text source.
Parses each as the corresponding type by calling the type on it. For
invalid or missing values, substitutes the value of calling the type
without arguments, or None if even that fails. May or may not
discard the remaining part of the line that contained the last
value. Use at your own risk!"""
...
from itertools import repeat
with open("f.txt" as f:
a, b, c = readWithDWIM(f, int, int, int))
xs = readWithDWIM(f, *repeat(float, b))
If there really is demand for these, they should be all over the web
already. I don't see why there should be demand. The simple case is
simple enough, more complex specifications are either still simple
enough to write from scratch, or too specific, or insane.
[...]
--
https://mail.python.org/mailman/listinfo/python-list
Re: SQLite
On 02/22/2016 02:44 AM, Chris Angelico wrote: On Mon, Feb 22, 2016 at 8:37 AM, Albert-Jan Roskam [1] wrote: IIRC, you have to do sudo apt-get install build-essential python-dev ... then re-compile python That may well work, but it's probably easier to work this way: sudo apt-get build-dep python3 That should grab all the compilation dependencies of the python3 package, which on Wheezy is a 3.2 IIRC. The deps haven't changed since then AFAIK. When you build, you should get a summary at the end that tells you about any modules that couldn't be built. The most common reason for that is missing deps. If you still have any after running the above, post here and we may be able to more directly advise. ChrisA That did not work because I did not install Python 3.5. with apt-get but downloaded the source and compiled myself. Thus apt-get does not have any information about the Python 3.5 installation. I got it to work following the hint by Albert-Jan Roskam. -- K.D.J. References Visible links 1. mailto:sjeik_ap...@hotmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: SQLite
On 02/21/2016 10:37 PM, Albert-Jan Roskam wrote: (Sorry for top posting) IIRC, you have to do sudo apt-get install build-essential python-dev ... then re-compile python > To: [1]python-list@python.org > From: [2]k.d.jant...@mailbox.org > Subject: SQLite > Date: Sun, 21 Feb 2016 18:11:18 +0100 > > Hello, > > I have downloaded Python3.5.1 as .targz, compiled it(configure, make,...) > and it works > (under Debian Wheezy AMD64) up to the moment I wanted to use SQLite. > > I get the following message: > === > jantzen@PC4:~$ python > Python 3.5.0 (default, Dec 2 2015, 14:16:16) > [GCC 4.7.2] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import sqlite3 > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python3.5/sqlite3/__init__.py", line 23, in > > from sqlite3.dbapi2 import * > File "/usr/local/lib/python3.5/sqlite3/dbapi2.py", line 27, in > from _sqlite3 import * > ImportError: No module named '_sqlite3' > === > > Obviously something is missing. > How do I solve the problem? Where do I find this module? > > Thanks for a hint. > -- > > K.D.J. > -- > [3]https://mail.python.org/mailman/listinfo/python-list Hello, thanks for your hint. That did it!!! -- K.D.J. References Visible links 1. mailto:python-list@python.org 2. mailto:k.d.jant...@mailbox.org 3. https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Considering migrating to Python from Visual Basic 6 for engineering applications
Note that you can continue using your existing vb6 code from python through COM using pywin32 or pythonnet, until you decide to rewrite it. -- https://mail.python.org/mailman/listinfo/python-list
