date:20171101

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

2017-11-01 Thread Guido van Rossum

Good write-ups. Though it sounds like idna should just be moved into the
stdlib. (Agreed that for things like this codecs are not a great idea.) We
have other things that must be updated whenever the Unicode standard
changes and it seems that doing this in feature releases is typically fine,
and treating it as a bugfix (so doing it in bugfix releases is acceptable)
sounds fine to me too. Unless it were so security-critical that we'd want
to enable users to do such updates to less-supported Pythons or on a
tighter schedule (which is one of the reasons requests can't go into the
stdlib -- they have their own cert store).

On Wed, Nov 1, 2017 at 7:36 PM, Nathaniel Smith  wrote:

> On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum  wrote:
> > Can you write 1-2 paragraphs with the argument for each?
> >
> > On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith  wrote:
> >> - lxml
>
> My impression (probably others are more knowledgeable) is that lxml
> has more or less replaced the stdlib 'xml' package as the de facto
> standard -- sort of similar to the urllib2/requests situation. AFAIK
> lxml has never been proposed for stdlib inclusion and I believe the
> fact that it's all in Cython would be a barrier even if the
> maintainers were amenable. But it might be helpful to our users to put
> a box at the top of the 'xml' docs suggesting people check out 'lxml',
> similar to the one on the urllib2 docs.
>
> >> - numpy
>
> Numpy's arrays are a foundational data structure and de facto
> standard, and would probably fit naturally in the stdlib semantically,
> but for a number of logistical/implementational reasons it doesn't
> make sense to merge. Probably doesn't make much difference whether
> python-dev "blesses" it or not in practice, since there aren't any
> real competitors inside or outside the stdlib; it'd more just be an
> acknowledgement of the status quo.
>
> >> - cryptography
>
> Conceptually, core cryptographic operations are the kind of
> functionality that you might expect to see in the stdlib, but the
> unique sensitivity of crypto code makes this a bad idea. Historically
> there have been a variety of basic crypto packages for Python, but at
> this point IIUC the other ones are all considered
> obsolete-and-potentially-dangerous and the consensus is everyone
> should move to 'cryptography', so documenting that in this PEP might
> help send people in the right direction.
>
> >> - idna
>
> This is a bit of a funny one. IDNA functionality is pretty fundamental
> -- you need it to do unicode<->bytes conversions on hostnames, so
> basically anyone doing networking needs it. Python ships with some
> built-in IDNA functionality (as the "idna" codec), but it's using an
> obsolete standard (IDNA2003, versus the current IDNA2008, see
> bpo-17305), and IIRC Christian thinks the whole codec-based design is
> the wrong approach... basically what we have in the stdlib has been
> broken for most of a decade and there doesn't seem to be anyone
> interested in fixing it. So... in the long run the stdlib support
> should either be fixed or deprecated. I'm not sure which is better.
> (The argument for deprecating it would be that IIUC you need to update
> the tables whenever a new unicode standard comes out, and since it's a
> networking thing you want to keep in sync with the rest of the world,
> which is easier with a standalone library. I don't know how much this
> matters in practice.) But right now, this library is just better than
> the stdlib functionality, and it wouldn't hurt to document that.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

2017-11-01 Thread Nathaniel Smith

On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum  wrote:
> Can you write 1-2 paragraphs with the argument for each?
>
> On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith  wrote:
>> - lxml

My impression (probably others are more knowledgeable) is that lxml
has more or less replaced the stdlib 'xml' package as the de facto
standard -- sort of similar to the urllib2/requests situation. AFAIK
lxml has never been proposed for stdlib inclusion and I believe the
fact that it's all in Cython would be a barrier even if the
maintainers were amenable. But it might be helpful to our users to put
a box at the top of the 'xml' docs suggesting people check out 'lxml',
similar to the one on the urllib2 docs.

>> - numpy

Numpy's arrays are a foundational data structure and de facto
standard, and would probably fit naturally in the stdlib semantically,
but for a number of logistical/implementational reasons it doesn't
make sense to merge. Probably doesn't make much difference whether
python-dev "blesses" it or not in practice, since there aren't any
real competitors inside or outside the stdlib; it'd more just be an
acknowledgement of the status quo.

>> - cryptography

Conceptually, core cryptographic operations are the kind of
functionality that you might expect to see in the stdlib, but the
unique sensitivity of crypto code makes this a bad idea. Historically
there have been a variety of basic crypto packages for Python, but at
this point IIUC the other ones are all considered
obsolete-and-potentially-dangerous and the consensus is everyone
should move to 'cryptography', so documenting that in this PEP might
help send people in the right direction.

>> - idna

This is a bit of a funny one. IDNA functionality is pretty fundamental
-- you need it to do unicode<->bytes conversions on hostnames, so
basically anyone doing networking needs it. Python ships with some
built-in IDNA functionality (as the "idna" codec), but it's using an
obsolete standard (IDNA2003, versus the current IDNA2008, see
bpo-17305), and IIRC Christian thinks the whole codec-based design is
the wrong approach... basically what we have in the stdlib has been
broken for most of a decade and there doesn't seem to be anyone
interested in fixing it. So... in the long run the stdlib support
should either be fixed or deprecated. I'm not sure which is better.
(The argument for deprecating it would be that IIUC you need to update
the tables whenever a new unicode standard comes out, and since it's a
networking thing you want to keep in sync with the rest of the world,
which is easier with a standalone library. I don't know how much this
matters in practice.) But right now, this library is just better than
the stdlib functionality, and it wouldn't hurt to document that.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Membership of infinite iterators

2017-11-01 Thread Nick Coghlan

On 2 November 2017 at 04:29, Koos Zevenhoven  wrote:

> SPOILER ALERT! At the moment, Nick's statement is in fact **always** true
> in **all** cases (at least when ignoring untypical cases and some
> inaccuracies in phrasing). Another question is whether the statement
> **should** be true at all.
> 
>
> PyErr_CheckSignals(), the function that checks for pending signals, now
> **implicitly** uses the strictest possible memory-order requirement
> (SEQ_CST) for checking the `is_tripped` flag, a value which can be used to
> peek whether there are any pending signals. This means that two threads
> that call PyErr_CheckSignals can't even **check** the value of that flag
> at the same time, and they'll have to wait for each other and whatever the
> CPU hardware implementation does for cache synchronzation.
>

Nice, that's deeper than I went - I just assumed there was an
interlocked_increment in there somewhere, without checking whether or not
there were any optimised code paths that managed to avoid that call :)


> From a correctness point of view, that is absolutely great: if
> PyErr_CheckSignals() is called, it is guaranteed to notice a new signal
> regardles of how small the number of picoseconds after the `is_tripped`
> flag has been set.  But is that really important? Do we really need things
> to be slowed down by this? And do we want people to "optimize" code by
> avoiding signal checking?
>

It isn't signal handling latency that's the concern here, it's potentially
missing signals. Consider the case where we have 3 threads: A, B, C. A is
the main thread that will actually handle signals, B and C both happened to
receive them. We'll also assume  the two threads receive *different*
signals (otherwise they'll potentially get collapsed into one notification
regardless).

The scenario we want to avoid is having one or both of the signals set, but
is_tripped cleared. With an interlocked check (where both 0->1 and 1->0 are
memory barriers), that clearly can't happen. I suspect you're right that
this could also be achieved with a less strict memory sync requirement on
the 0->1 check, but that's much harder to reason about than simply limiting
the number of iterations we make through an iterator consumption loop
before checking for signals.


> The signals won't be caught until checked anyway, and as we've seen, one
> solution is to use a counter to determine if enough time has passed that
> another check for potential signals should happen. That does, however,
> raise the question of who should own the counter, and if it's OK for
> multiple threads to use the same counter. If yes, then would we be able to
> avoid slow atomic decrements (or increments)?
>
> But another solution might be to make a less strict but almost equivalent
> functionality with much less overhead. Something like this in a header file:
>
> static inline int PyErr_PROBE_SIGNALS() {
> static int volatile *flag = (int volatile *) _tripped;
> if (*flag) {
> return PyErr_CheckSignals();
> }
> else {
> return 0;
> }
> }
>
> Here, the flag is casted to volatile to force a check to happen each time
> from memory/cache. However, to make it more standard and to make sure it
> works with all compilers/platforms, it might be better to, instead of
> *flag, use an atomic load with "MEMORY_ORDER_RELAXED". Another thing is
> that `is_tripped`, which is currently declared as static inside
> signalmodule.c [4], should somehow be exposed in the API to make it
> available for the inline function in headers.
>
> This solution might be fast enough for a lot of cases, although still
> slightly slower than the counter approach, at least if the counter approach
> would completely avoid per-iteration memory access by requiring each
> function to own the counter that is used for this.
>

One thing I like about a nested inner loop is that it's easy to understand
the rationale for it *without* knowing any of the details of how memory
synchronisation works, as it's just:

- we want to check for signals regularly so the latency in interrupt
handling isn't too high
- PyErr_CheckSignals() is too expensive to call on every iteration
- so we only check every N iterations

Memory synchronisation then only comes up if someone asks why
"PyErr_CheckSignals" is expensive to call. And while it wouldn't surprise
at all if you're right and there are ways to make that call cheaper,
they're still never going to be cheaper than explicitly reducing the
frequency with which it is called.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Terry Reedy


On 11/1/2017 5:47 PM, Terry Reedy wrote:
When pip installs a package into site_packages, does it at any point run 
  package-specific installation code?  setup.py?  More specifically, can 
pip install an IDLE extension. If so, I think installing pipgui should 
add 'x_pipgui.py' to idlelib, if it exists, and add a section to 
idlelib/config-extension.def.  Using the existing extension mechanism 
would be an alternative to patching IDLE to conditionally add pipgui to 
some menu.


I meant to add that using the existing extension mechanism would allow 
pipgui to be run from 3.5 IDLE or even earlier, depending on how it is 
coded and the tcl/tk installed on the machine.  Pipgui really needs ttk, 
which requires tcl/tk >= 8.5, but that has been true on Python Windows 
installs at least since 2.7.


--
Terry Jan Reedy


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Terry Reedy


On 11/1/2017 3:06 AM, Terry Reedy wrote:

On 11/1/2017 1:25 AM, Nick Coghlan wrote:
On 1 November 2017 at 08:50, Terry Reedy > wrote:


    In April 2016, after posting the idea to pydev list and getting 'go
    ahead's from Nick Coughlin and someone else, with no negatives, I
    approved Upendra Kumar's GSOC proposal to write a pip gui.  This was
    https://bugs.python.org/issue27051
    .  On June 20, Ned Deily and
    Nick Coughlin vetoed adding a pip gui anywhere in the stdlib since
    it depended on something not in the stdlib, and perhaps for other
    reasons I don't fully understand.


Clarifying the objection here (since the linked issue is a fairly long 
one): what I'm against is tightly coupling the pip-gui development & 
release process to the CPython development & release process when we 
don't have any compelling reason to do so.


Thank you for clarifying. This is a reason that I did not understand ;-).

Given an independently released pip-gui on PyPI (with its own version 
numbering and release cadence), then I'd be +1 on bundling that as an 
optional IDLE addon, ensurepip style.


I already agree in my second response on this thread that pip gui should 
be on pypi.


A pip gui would be a pip add-on, not an IDLE add-on, just as turtledemo 
is a turtle add-on even though one can invoke it from the IDLE menu -- 
as long as turtle and turtledemo are installed.


Issue 27051, write pip gui, is properly a separate issue from 
https://bugs.python.org/issue23551, start a pip gui, if and when 
available, from IDLE menu.  I am against 'tightly coupling' something to 
IDLE when there is 'no compelling reason to do so'.


When pip installs a package into site_packages, does it at any point run 
 package-specific installation code?  setup.py?  More specifically, can 
pip install an IDLE extension. If so, I think installing pipgui should 
add 'x_pipgui.py' to idlelib, if it exists, and add a section to 
idlelib/config-extension.def.  Using the existing extension mechanism 
would be an alternative to patching IDLE to conditionally add pipgui to 
some menu.


--
Terry Jan Reedy


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Hello from a new lurker

2017-11-01 Thread Ethan Furman


On 11/01/2017 10:49 AM, brent bejot wrote:


The python-dev website said I should introduce myself before lurking around for 
a while.  So hi!  I've been using python
for 7 or so years now, live on the east coast of the U.S., grew up in Nebraska, 
and occasionally play the tuba and
table-top games (through rarely together).  I'm looking forward to hearing 
where the language is going and hope I can be
of use after I lurk in the shadows for a while.


Welcome!

Enjoy your time lurking.  When new ideas come up, feel free to comment if that 
idea will help or hinder you.

--
~Ethan~

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Hello from a new lurker

2017-11-01 Thread brent bejot

Hello everyone,

The python-dev website said I should introduce myself before lurking around
for a while.  So hi!  I've been using python for 7 or so years now, live on
the east coast of the U.S., grew up in Nebraska, and occasionally play the
tuba and table-top games (through rarely together).  I'm looking forward to
hearing where the language is going and hope I can be of use after I lurk
in the shadows for a while.

Cheers,
Brent
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Wes Turner

On Wednesday, November 1, 2017, Steve Barnes  wrote:

>
>
> On 01/11/2017 06:54, Wes Turner wrote:
> > Suggestions to help to minimize unnecessary logged bandwidth use and
> > even work with a closed loop LAN:
> >
> > This reads from the filesystem:
> >
> >  import requests
> >
> > This would read from the PyPi service over the network bandwidth:
> >
> > #!pip install -U requests
> > #%run pip install -U requests
> >
> > #pip('install -U requests')
> >
> > This doesn't work because you SHOULD restart the interpreter after
> > running pip (because imports are cached):
> >
> >  import requests
> >  !pip install -U requests
> >  import requests
> >
> > Some tips on running educational environments for beginners (optionally
> > in a lab):
> >
> 
>
> One tip that I have used when teaching python in a closed, (sometimes
> internet free environment), was to pre-prepare by, in an on-line
> environment:
>
>   1. Create a virtual environment with the version of Python that I am
> going to be teaching on the target platform
>   2. Activate that environment
>   3. Ensure that I am On-line
>   4. Download the pip install packages that I know I will need by using
> `pip download` to download but not install the packages, ideally using
> the -r requirements.txt syntax, (plus any windows specific builds from
> Christoph Gohlke's site).
>   5. Go Off-line and run pip install with the downloaded package - if I
> hit any errors due to packages having unspecified dependencies add those
> to the requirements list and repeat from 3. (While I am at it I often
> log an issue with the package maintainer).
>   6. A fast, personal, run through my lesson plan to ensure that I
> haven't missed anything.
>
> I normally also download a few goodies that might not be essential to
> the lesson but that can act as a teaser for the more interested students.
>
> At the start of the first lesson I give the students the downloaded
> packages directory, usually on a USB key, and get them to pip install
> them while explaining the difference between local and on-line
> installation.


That works. You could also host the packages with devpi or just a static
HTTP server.

As a transparent cache, if you test your ``requirements.txt`` with each
OS/CPU_architecture combination, DevPi will store and serve each package to
everyone at once.

So do you do (in a virtualenv):

  pip install --download ./dir -r requirements.txt

And then

  pip install --no-index --find-links ./dir -r requirements.text

Like in the pip docs?

https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages

DevPi can host package documentation.
https://devpi.net/docs/devpi/devpi/stable/+doc/userman/devpi_packages.html#uploading-documentation
*

GitLab Pages can host package documentation and any static HTML pages (e.g.
the CPython docs) alongside **the source**
https://docs.gitlab.com/ce/user/project/pages/index.html

https://github.com/python/cpython/tree/3.6/Doc


ReadTheDocs can host Sphinx docs and can be run in a Docker container for
an internet-free LAN:
https://github.com/rtfd/readthedocs-docker-images


>
> I know that I could save having to get the students to run pip by
> packaging up the virtual environment as a portable, or using by
> pyInstaller, but having them run pip on the local downloads gives me a
> chance to explain how to do it in the wild.


AFAIU, YMMV with ``virtualenv --relocatable``; and it certainly doesn't do
all combinations of OS and processor architecture:

https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable

itertools.combinations(
  ['win', 'mac', 'lin', '', '686', 'x86_64', 'ARM'], 2)


>
> BTW while Docker is great for this it is a whole other learning
> experience, (plus getting it running with some corporate security &
> anti-virus can be quite a challenge).


Yeah, hosting Docker containers with Kubernetes across a couple extra
workstations is initially more work than creating a few USB keys and
waiting for everyone's virtualenvs to converge (with ``$ pip install``,
``!PIP_INDEX='' pip install``, and now a new pip GUI button press sequence).


>
> These USB keys are often re-used by other co-workers as a getting
> started or after my computer got changed/re-imaged starting point.


Wouldn't it be create if, just like oldschool floppies, there was a way to
break off the sliding write protect tab of a USB stick?

https://en.wikipedia.org/wiki/USB_flash_drive_security#Malware_infections

CD-ROM and DVD-ROM drives seem to be more and more scare these days.


>
> --
> Steve (Gadget) Barnes
> Any opinions in this message are my personal opinions and do not reflect
> those of my employer.
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org 
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Wes Turner

On Wednesday, November 1, 2017, Terry Reedy  wrote:

> On 11/1/2017 2:54 AM, Wes Turner wrote:
>
> A Tk GUI for pip would need to frustratingly duplicate ~/.bash_history and
>> up-arrow to get the previous command.
>>
>
> No it wouldn't, as the user would not be directly issuing pip commands. In
> any case, it would be trivial to keep a list of the pip commands submitted
> behind the gui facade and even to have an optional window to display them.

Now that there's a ``pip freeze`` and a ``conda env export``, it seems like
it's much easier to determine what one just did to their {system, --user,
virtualenv, conda root, or condaenv} without having a list of commands to
review (ala .bash_history) or filesystem diff's. Would such a GUI handle
these cases and also virtualenv create-and-restart?

Bash can be installed in Windows.

>
> IDLE sucks.
>>
>
> Why pollute your post with stupid flame bait?  In many respects, IDLE's
> Shell is unequivocally better than interactive python in a line-oriented
> shell like Command Prompt.  Ironically, given your comment above, one of
> its improvements is keeping a history of *statements* rather than a history
> of lines.  This makes it possible to retrieve and edit a whole command
> (statement) at once, rather than in pieces (lines).

Spyder is an IDE with a command prompt (shell window) and a python prompt
(python window, ipython window).

>
> --
> Terry Jan Reedy
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Steve Barnes

On 01/11/2017 06:54, Wes Turner wrote:
> Suggestions to help to minimize unnecessary logged bandwidth use and 
> even work with a closed loop LAN:
> 
> This reads from the filesystem:
> 
>      import requests
> 
> This would read from the PyPi service over the network bandwidth:
> 
>     #!pip install -U requests
>     #%run pip install -U requests
> 
>     #pip('install -U requests')
> 
> This doesn't work because you SHOULD restart the interpreter after 
> running pip (because imports are cached):
> 
>      import requests
>      !pip install -U requests
>      import requests
> 
> Some tips on running educational environments for beginners (optionally 
> in a lab):
> 

One tip that I have used when teaching python in a closed, (sometimes 
internet free environment), was to pre-prepare by, in an on-line 
environment:

  1. Create a virtual environment with the version of Python that I am 
going to be teaching on the target platform
  2. Activate that environment
  3. Ensure that I am On-line
  4. Download the pip install packages that I know I will need by using 
`pip download` to download but not install the packages, ideally using 
the -r requirements.txt syntax, (plus any windows specific builds from 
Christoph Gohlke's site).
  5. Go Off-line and run pip install with the downloaded package - if I 
hit any errors due to packages having unspecified dependencies add those 
to the requirements list and repeat from 3. (While I am at it I often 
log an issue with the package maintainer).
  6. A fast, personal, run through my lesson plan to ensure that I 
haven't missed anything.

I normally also download a few goodies that might not be essential to 
the lesson but that can act as a teaser for the more interested students.

At the start of the first lesson I give the students the downloaded 
packages directory, usually on a USB key, and get them to pip install 
them while explaining the difference between local and on-line installation.

I know that I could save having to get the students to run pip by 
packaging up the virtual environment as a portable, or using by 
pyInstaller, but having them run pip on the local downloads gives me a 
chance to explain how to do it in the wild.

BTW while Docker is great for this it is a whole other learning 
experience, (plus getting it running with some corporate security & 
anti-virus can be quite a challenge).

These USB keys are often re-used by other co-workers as a getting 
started or after my computer got changed/re-imaged starting point.

-- 
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect 
those of my employer.

---
This email has been checked for viruses by AVG.
http://www.avg.com

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Terry Reedy


On 11/1/2017 2:54 AM, Wes Turner wrote:

A Tk GUI for pip would need to frustratingly duplicate ~/.bash_history 
and up-arrow to get the previous command.


No it wouldn't, as the user would not be directly issuing pip commands. 
In any case, it would be trivial to keep a list of the pip commands 
submitted behind the gui facade and even to have an optional window to 
display them.



IDLE sucks.


Why pollute your post with stupid flame bait?  In many respects, IDLE's 
Shell is unequivocally better than interactive python in a line-oriented 
shell like Command Prompt.  Ironically, given your comment above, one of 
its improvements is keeping a history of *statements* rather than a 
history of lines.  This makes it possible to retrieve and edit a whole 
command (statement) at once, rather than in pieces (lines).


--
Terry Jan Reedy

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Terry Reedy


On 11/1/2017 1:25 AM, Nick Coghlan wrote:
On 1 November 2017 at 08:50, Terry Reedy 
> wrote:


In April 2016, after posting the idea to pydev list and getting 'go
ahead's from Nick Coughlin and someone else, with no negatives, I
approved Upendra Kumar's GSOC proposal to write a pip gui.  This was
https://bugs.python.org/issue27051
.  On June 20, Ned Deily and
Nick Coughlin vetoed adding a pip gui anywhere in the stdlib since
it depended on something not in the stdlib, and perhaps for other
reasons I don't fully understand.


Clarifying the objection here (since the linked issue is a fairly long 
one): what I'm against is tightly coupling the pip-gui development & 
release process to the CPython development & release process when we 
don't have any compelling reason to do so.


Thank you for clarifying. This is a reason that I did not understand ;-).

Given an independently released pip-gui on PyPI (with its own version 
numbering and release cadence), then I'd be +1 on bundling that as an 
optional IDLE addon, ensurepip style.


I already agree in my second response on this thread that pip gui should 
be on pypi.


A pip gui would be a pip add-on, not an IDLE add-on, just as turtledemo 
is a turtle add-on even though one can invoke it from the IDLE menu -- 
as long as turtle and turtledemo are installed.


Issue 27051, write pip gui, is properly a separate issue from 
https://bugs.python.org/issue23551, start a pip gui, if and when 
available, from IDLE menu.  I am against 'tightly coupling' something to 
IDLE when there is 'no compelling reason to do so'.


--
Terry Jan Reedy


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] install pip packages from Python prompt

2017-11-01 Thread Wes Turner

Suggestions to help to minimize unnecessary logged bandwidth use and even
work with a closed loop LAN:

This reads from the filesystem:

import requests

This would read from the PyPi service over the network bandwidth:

   #!pip install -U requests
   #%run pip install -U requests

   #pip('install -U requests')

This doesn't work because you SHOULD restart the interpreter after running
pip (because imports are cached):

import requests
!pip install -U requests
import requests

Some tips on running educational environments for beginners (optionally in
a lab):

- IPython
  - requests?  # docstring
  - requests?? # source
  - !pydoc ipython
  - %run pydoc ipython
- Spyder includes an IPython terminal
  (``conda install spyder`` even installs Qt)
- Just use conda (and docker and binder)
- Aggressively cache network accesses (and/or just block internet access
entirely)
  - PIP_INDEX="https://pip.local/simple/;
  - DevPi can transparently proxy cache packages from pypi (and then work
without internet access)
  - a local httpbin is easy to host within the LAN

- Binder can provision Docker containers with conda and everything already
installed (With Kubernetes and JupyterHub (and OAuth))
  - Each person gets their own container
  - Jupyter has a web terminal built in
  - You can also just run the Docker container locally

#!pip install ipython scipy devpi certbot
# Some back of the napkin calculations:
# 30 users run a script which runs 2 pip install requests at least 5 times.
# (These should be at the top)

These Docker containers are ready to go with everything you need for a
lesson:
https://github.com/jupyter/docker-stacks

You don't need this bash shell lesson if all you need to do is ``!pip
install`` with IPython:
https://swcarpentry.github.io/shell-novice/

A Tk GUI for pip would need to frustratingly duplicate ~/.bash_history and
up-arrow to get the previous command.
(Qt and GTK are not consistently installed; though ``conda install spyder``
does install Qt (which is very accessible))

IDLE sucks.

https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html

https://github.com/kennethreitz/httpbin/blob/master/Dockerfile

These should pretty much get you started:

https://github.com/jrjohansson/scientific-python-lectures

https://github.com/jrjohansson/scientific-python-lectures/blob/master/Lecture-0-Scientific-Computing-with-Python.ipynb

https://github.com/jrjohansson/scientific-python-lectures/blob/master/Lecture-1-Introduction-to-Python-Programming.ipynb

https://github.com/jrjohansson/scientific-python-lectures/blob/master/Lecture-7-Revision-Control-Software.ipynb


GitLab and Mattermost work with an offline LAN:

https://docs.gitlab.com/omnibus/README.html
https://docs.gitlab.com/omnibus/docker/README.html#run-the-image

Binder can pull images from a GitLab Docker Container Registry:
https://docs.gitlab.com/ce/user/project/container_registry.html

These suggestions should help to minimize unnecessary logged bandwidth use
and even work with a closed loop LAN.

See: "#!pip install ipython scipy devpi certbot" in the middle of this
email.

On Tuesday, October 31, 2017, Wes Turner  wrote:

> You could teach them subprocess and os command injection safety from the
> start:
>
> ```python
> import subprocess
> import sys
> cmd = [sys.executable, -m', 'pip', 'install', '-r',
> 'psfblessed-requirements.txt'])
> retcode = subprocess.check_call(cmd)
> assert retcode == 0
> ```
>
> (Because shell=True is dangerous and str.split is dangerous):
>
> ```python
> filename = "'/etc/ passwd' ; shutdown -r now"
> cmd = ("cat '%s'" % filename)
> cmd
> # "cat ''/etc/ passwd'' ; shutdown -r now"
>
> cmd.split()
> # ["'", '/etc', 'passwd', ';', 'shutdown', '-r', 'now']
>
> shlex.split(cmd)
> # ['cmd', '', '/etc', 'passwd', ';', 'shutdown', '-r', 'now']
> ```
>
> (Sarge solves for a number of additional cases beyond shlex.split (empty
> string should be '', 'already-quoted commands')
>
> https://sarge.readthedocs.io/en/latest/overview.html#why-
> not-just-use-subprocess
>
>  https://en.wikipedia.org/wiki/Code_injection#Shell_injection
>
> https://sarge.readthedocs.io/en/latest/internals.html#how-
> shell-quoting-works
>
> Of course, we're programmers and our input is not untrusted, so shell=True
> without any string operations is not as dangerous.
>
>
> On Tuesday, October 31, 2017, Wes Turner  > wrote:
>
>> You could teach them subprocess and os command injection safety from the
>> start:
>>
>> ```python
>> import subprocess
>> import sys
>> cmd = [sys.executable, -m', 'pip', 'install', '-r',
>> 'psfblessed-requirements.txt'])
>> retcode = subprocess.check_call(cmd)
>> assert retcode == 0
>> ```
>>
>> (Because shell=True is dangerous)
>>
>> On Tuesday, October 31, 2017, Terry Reedy  wrote:
>>
>>> On 10/31/2017 12:21 PM, Ivan Levkivskyi wrote:
>>>
 I think it was proposed several times

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

Re: [Python-ideas] Membership of infinite iterators

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] Hello from a new lurker

[Python-ideas] Hello from a new lurker

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] install pip packages from Python prompt

Re: [Python-ideas] install pip packages from Python prompt

13 matches

Site Navigation

Mail list logo

Footer information