Re: [Python-Dev] Deprecating float.is_integer()

[Tim Peters]

[Kirill Balunov ]
> ...
> In spite of the fact that the pronouncement has
> already been made, there may still be an opportunity to influence this
> decision.

That's not really how this works.  Guido has been doing this for
decades, and when he Pronounces he's done with it :-)


> I do not think that this is only a matter of choice, how this
> functionality will be accessed through a method or function, in fact these
> highly specialized methods heavily pollute the API

"Heavily"?  Seems oversold.


> and open the door for persistent questions.

That's a door that can never be closed, no matter what.


> Given the frequency and  activity of using this
> `.is_integer` method the deprecation of this method is unlikely to greatly
> affect someone. (for `as_integer_ratio` I think the bar is higher).
> Summarizing this thread it seems to me that with deprecation of `is_integer`
> method and with addition of `is_integer` function in math module will make
> everyone happy:

Not at all, but that's already been explained.  Deprecation is
_serous_ business:  it's not only the presumably relative handful of
direct users who are directly annoyed, but any number of worldwide web
pages, blogs, books, papers, slides, handouts, message boards ... that
so much as mentioned the now-deprecated feature.  The language
implementation is the tiniest part of what's affected, yet is the
_only_ part we (Python developers) can repair.

Deprecation really requires that something is a security hole that
can't be repaired, impossible to make work as intended, approximately
senseless, or is superseded by a new way to accomplish a thing that's
near-universally agreed to be vastly superior.  Maybe others?
Regardless, they're all "really big deals".

The "harm" done by keeping these methods seems approximately
insignificant.  Serhiy certainly found examples where uses made no
good sense, but that's _common_ among floating-point features.  For
example, here's a near-useless implementation of Newton's method for
computing square roots:

def mysqrt(x):
guess = x / 2.0
while guess ** 2 != x:
guess = (guess + x / guess) / 2.0
return guess

And here I'll use it:

>>> mysqrt(25.0)
5.0
>>> mysqrt(25.2)
5.019960159204453

Works great!  Ship it :-)

>>> mysqrt(25.1)

Oops.  It just sits there, consuming cycles.

That's because there is no IEEE double x such that x*x == 25.1.  While
that's not at all obvious, it's true.  Some people really have argued
to deprecate (in)equality testing of floats because of "things like
that", but that's fundamentally nuts.  We may as well remove floats
entirely then.

In short, that an fp feature can be misused, and _is_ misused, is no
argument for deprecating it.  If it can _only_ be misused, that's
different, but that doesn't apply to is_integer.

That someone - or even almost everyone - is merely annoyed by seeing
an API they have no personal use for doesn't get close to "really big
deal".  The time to stop it was before it was added.


> PROS:
> ...
> 5. Make everyone happy and stop this thread :)

This thread ended before you replied to it - I'm just a ghost haunting
its graveyard to keep you from feeling ignored  -)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better support for consuming vendored packages

[Barry Warsaw]

On Mar 22, 2018, at 12:33, Oleg Broytman  wrote:
> 
> On Thu, Mar 22, 2018 at 12:30:02PM -0700, Barry Warsaw  
> wrote:
>> Developers are mostly going to use pip, and maybe a requirements.txt,
> 
>   +virtual envs to avoid problems with global site-packages.

Yep, that was implied but of course it’s better to be explicit. :)

>   IMO virtualenv for development and frozen app for distribution solve
> the problem much better than vendoring.

+1
-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better support for consuming vendored packages

[Oleg Broytman]

On Thu, Mar 22, 2018 at 12:30:02PM -0700, Barry Warsaw  wrote:
> Developers are mostly going to use pip, and maybe a requirements.txt,

   +virtual envs to avoid problems with global site-packages.

   IMO virtualenv for development and frozen app for distribution solve
the problem much better than vendoring.

> Cheers,
> -Barry

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better support for consuming vendored packages

[Barry Warsaw]

On Mar 22, 2018, at 09:58, Gregory Szorc  wrote:
> 
> Not all consumers of Python packages wish to consume Python packages in the 
> common `pip install ` + `import ` manner. Some Python 
> applications may wish to vendor Python package dependencies such that known 
> compatible versions are always available.

It’s important to understand how painful vendoring is to some downstream 
consumers.  Debian is a good example.  There we often have to go through a lot 
of hoops to unvendor packages, both for policy and for good distribution 
practices.  The classic example is a security vulnerability in a library.  It’s 
the distro’s responsibility to fix that, but in the face of vendored 
dependencies, you can’t just patch the system package.  Now you also have to 
hunt down all the vendored versions and figure out if *they’re* vulnerable, 
etc.  It certainly doesn’t help that you can easily have vendored libraries 
vendoring their own dependencies.  I think I found one application in Debian 
once that had like 4 or 5 versions of urllib3 inside it!

You mention dependency hell for downstream consumers like a Linux distro, but 
this type of integration work is exactly the job of a distro.  They have to 
weigh the health and security of all the applications and libraries they 
support, so it doesn’t bother me that it’s sometimes challenging to work out 
the right versions of library dependencies.  It bothers me a lot that I have to 
(sometimes heavily) modify packages to devendorize dependencies, especially 
because it’s not always clearly evident that that has happened.

That said, I completely understand the desire for application and library 
authors to pin their dependency versions.  We’ve had some discussions in the 
past (not really leading anywhere) about how to satisfy both communities.  I 
definitely don’t go so far as to discourage global imports, and I definitely 
don’t like vendoring all your dependencies.  For applications distributed 
outside of a distro, there are lots of options, from zip apps (e.g. pex) to 
frozen binaries, etc.

Developers are mostly going to use pip, and maybe a requirements.txt, so I 
think that use case is well covered.  Downstream consumers need to be able to 
easily devendorize, but I think ultimately, the need to vendorize just points 
to something more fundamental missing from Python’s distribution and import 
system.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better support for consuming vendored packages

[Gregory Szorc]

On 3/22/2018 10:48 AM, Oleg Broytman wrote:
> Hi!
> 
> On Thu, Mar 22, 2018 at 09:58:07AM -0700, Gregory Szorc 
>  wrote:
>> Not all consumers of Python packages wish to consume Python packages in the
>> common `pip install `
> 
> IMO `pip` is for developers. To package and distribute end-user
> applications there are rpm, dpkg/deb, PyInstaller, cx_Freeze, py2exe
> (+ installer like NSIS or InnoSetup), py2app, etc...
> 
> Most of them pack a copy of Python interpreter and necessary parts
> of stdlib, so there is no problem with `sys.path` and wrong imports.

Yes, there are tools to create standalone packages. Some even bundle a
Python install so the execution environment is more deterministic. These
are great ways to distribute Python applications!

However, if you are a Python application that is maintained by your
distribution's package manager, you pretty much must use a Python
installed by the system package manager. And that leaves us with the
original problem of an undefined execution environment. So packaging
tools for standalone Python applications only work if you control the
package distribution channel. If you are a successful Python application
that is packaged by your distro, you lose the ability to control your
own destiny and must confront these problems for users who installed
your application through their distro's package manager. i.e. the cost
of success for your Python application is a lot of pain inflicted by
policies of downstream packagers.

Also, not vendoring dependencies puts the onus on downstream packagers
to deal with those dependencies. There can be package version conflicts
between various packaged Python applications ("dependency hell").
Vendoring dependencies under application-local package names removes the
potential for version conflicts.

It's worth noting that some downstream packagers do insist on unbundling
dependencies. So they may get stuck with work regardless. But if you
vendor dependencies, a downstream packager is at least capable of
packaging a Python application without having to deal with "dependency
hell."

Maybe what I'm asking for here is import machinery where an application
can forcefully limit or influence import mechanisms for modules in a
certain package. But this seems difficult to achieve given the
constraint of a single, global modules namespace (`sys.modules`) per
interpreter.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Kirill Balunov]

2018-03-22 19:47 GMT+03:00 Tim Peters :

>
> > Is this functionality so often used and practical to be a method of
> float,
> > int, ..., and not just to be an auxiliary function?
> >
> > p.s.: The same thoughts about `as_integer_ratio` discussion.
>
> I would have added them as functions in the `math` module instead.
> perhaps supported by dunder methods (__as_integer_ratio__,
> __is_integer__).  But that's not what happened, and whether or not
> they have double underscores on each end doesn't really make all that
> much difference except to dedicated pedants ;-)
>

Yes, this was my point. In spite of the fact that the pronouncement has
already been made, there may still be an opportunity to influence this
decision. I do not think that this is only a matter of choice, how this
functionality will be accessed through a method or function, in fact these
highly specialized methods heavily pollute the API and open the door for
persistent questions. Given the frequency and  activity of using this
`.is_integer` method the deprecation of this method is unlikely to greatly
affect someone. (for `as_integer_ratio` I think the bar is
higher). Summarizing this thread it seems to me that with deprecation of
`is_integer` method and with addition of `is_integer` function in math
module will make everyone happy:

PROS:

1. Those who do not like this method, and do not want to see it as a
redundant part of `int`, ... will be happy
2. Those who need it will have this functionality through math module
3. Compatible packages do not have to quack louder
4. Cleaner API (no need to add this through numeric tower)
5. Make everyone happy and stop this thread :)

CONS:

1. Backward incompatible change

I do not want to restart this topic, but I think that there is an
opportunity for improvement that can be missed.

With kind regards,
-gdg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better support for consuming vendored packages

[Oleg Broytman]

Hi!

On Thu, Mar 22, 2018 at 09:58:07AM -0700, Gregory Szorc 
 wrote:
> Not all consumers of Python packages wish to consume Python packages in the
> common `pip install `

IMO `pip` is for developers. To package and distribute end-user
applications there are rpm, dpkg/deb, PyInstaller, cx_Freeze, py2exe
(+ installer like NSIS or InnoSetup), py2app, etc...

Most of them pack a copy of Python interpreter and necessary parts
of stdlib, so there is no problem with `sys.path` and wrong imports.

> Gregory

Oleg.
-- 
 Oleg Broytmanhttp://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Better support for consuming vendored packages

[Gregory Szorc]

 I'd like to start a discussion around practices for vendoring package
dependencies. I'm not sure python-dev is the appropriate venue for this
discussion. If not, please point me to one and I'll gladly take it there.

I'll start with a problem statement.

Not all consumers of Python packages wish to consume Python packages in the
common `pip install ` + `import ` manner. Some Python
applications may wish to vendor Python package dependencies such that known
compatible versions are always available.

For example, a Python application targeting a general audience may not wish
to expose the existence of Python nor want its users to be concerned about
Python packaging. This is good for the application because it reduces
complexity and the surface area of things that can go wrong.

But at the same time, Python applications need to be aware that the Python
environment may contain more than just the Python standard library and
whatever Python packages are provided by that application. If using the
system Python executable, other system packages may have installed Python
packages in the system site-packages and those packages would be visible to
your application. A user could `pip install` a package and that would be in
the Python environment used by your application. In short, unless your
application distributes its own copy of Python, all bets are off with
regards to what packages are installed. (And even then advanced users could
muck with the bundled Python, but let's ignore that edge case.)

In short, `import X` is often the wild west. For applications that want to
"just work" without requiring end users to manage Python packages, `import
X` is dangerous because `X` could come from anywhere and be anything -
possibly even a separate code base providing the same package name!

Since Python applications may not want to burden users with Python
packaging, they may vendor Python package dependencies such that a known
compatible version is always available. In most cases, a Python application
can insert itself into `sys.path` to ensure its copies of packages are
picked up first. This works a lot of the time. But the strategy can fall
apart.

Some Python applications support loading plugins or extensions. When
user-provided code can be executed, that code could have dependencies on
additional Python packages. Or that custom code could perform `sys.path`
modifications to provide its own package dependencies. What this means is
that `import X` from the perspective of the main application becomes
dangerous again. You want to pick up the packages that you provided. But
you just aren't sure that those packages will actually be picked up. And to
complicate matters even more, an extension may wish to use a *different*
version of a package from what you distribute. e.g. they may want to adopt
the latest version that you haven't ported to you or they may want to use
an old versions because they haven't ported yet. So now you have the
requirements that multiple versions of packages be available. In Python's
shared module namespace, that means having separate package names.

A partial solution to this quagmire is using relative - not absolute -
imports. e.g. say you have a package named "knights." It has a dependency
on a 3rd party package named "shrubbery." Let's assume you distribute your
application with a copy of "shrubbery" which is installed at some packages
root, alongside "knights:"

  /
  /knights/__init__.py
  /knights/ni.py
  /shrubbery/__init__.py

If from `knights.ni` you `import shrubbery`, you /could/ get the copy of
"shrubbery" distributed by your application. Or you could pick up some
other random copy that is also installed somewhere in `sys.path`.

Whereas if you vendor "shrubbery" into your package. e.g.

  /
  /knights/__init__.py
  /knights/ni.py
  /knights/vendored/__init__.py
  /knights/vendored/shrubbery/__init__.py

Then from `knights.ni` you `from .vendored import shrubbery`, you are
*guaranteed* to get your local copy of the "shrubbery" package.

This reliable behavior is highly desired by Python applications.

But there are problems.

What we've done is effectively rename the "shrubbery" package to
"knights.vendored.shrubbery." If a module inside that package attempts an
`import shrubbery.x`, this could fail because "shrubbery" is no longer the
package name. Or worse, it could pick up a separate copy of "shrubbery"
somewhere else in `sys.path` and you could have a Frankenstein package
pulling its code from multiple installs. So for this to work, all
package-local imports must be using relative imports. e.g. `from . import
x`.

The takeaway is that packages using relative imports for their own modules
are much more flexible and therefore friendly to downstream consumers that
may wish to vendor them under different names. Packages using relative
imports can be dropped in and used, often without source modifications.
This is a big deal, as downstream consumers don't want to be
modifying/forking 

Re: [Python-Dev] Deprecating float.is_integer()

[Tim Peters]

[Kirill Balunov ]
> I apologize that I get into the discussion. Obviously in some situations it
> will be useful to check that a floating-point number is integral, but from
> the examples given it is clear that they are very rare. Why the variant with
> the inclusion of this functionality into the math module was not considered
> at all.

Nobody here really discussed the history, and I don't know.  The
questions here have been about what to do given that `is_integer` and
`as_integer_ratio` are _already_ advertised (public) methods on some
numeric types.


> If the answer is - consistency upon the numeric tower - will it go
> for complex type and what will it mean (there can be two point of views)?

I haven't seen anyone suggest either method be added to Complex.
There are lots of methods that don't show up in the tower before
hitting Real.  For example, given that Complex doesn't support
__float__, it would be bizarre if it _did_ support as_integer_ratio.


> Is this functionality so often used and practical to be a method of float,
> int, ..., and not just to be an auxiliary function?
>
> p.s.: The same thoughts about `as_integer_ratio` discussion.

I would have added them as functions in the `math` module instead.
perhaps supported by dunder methods (__as_integer_ratio__,
__is_integer__).  But that's not what happened, and whether or not
they have double underscores on each end doesn't really make all that
much difference except to dedicated pedants ;-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Robert Smallshire]

In the PR which implements is_integer() for int, the numeric tower, and
Decimal I elected not to implement it for Complex or complex. This was
principally because complex instances, even if they have an integral real
value, are not convertible to int and it seems reasonable to me that any
number for which is_integer() returns True should be convertible to int
successfully, and without loss of information.

  >>> int(complex(2, 0))

  Traceback (most recent call last):

File "", line 1, in 

  TypeError: can't convert complex to int


There could be an argument that a putative complex.is_integral() should
therefore return False, but I expect that would get even less support than
the other suggestions in these threads.

*Robert Smallshire | *Managing Director
*Sixty North* | Applications | Consulting | Training
r...@sixty-north.com | T +47 63 01 04 44 | M +47 924 30 350
http://sixty-north.com

On 22 March 2018 at 10:51, Kirill Balunov  wrote:

> I apologize that I get into the discussion. Obviously in some situations
> it will be useful to check that a floating-point number is integral, but
> from the examples given it is clear that they are very rare. Why the
> variant with the inclusion of this functionality into the math module was
> not considered at all. If the answer is - consistency upon the numeric
> tower - will it go for complex type and what will it mean (there can be two
> point of views)?
>
> Is this functionality so often used and practical to be a method of float,
> int, ..., and not just to be an auxiliary function?
>
> p.s.: The same thoughts about `as_integer_ratio` discussion.
>
> With kind regards,
> -gdg
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> rob%40sixty-north.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Kirill Balunov]

 I apologize that I get into the discussion. Obviously in some situations
it will be useful to check that a floating-point number is integral, but
from the examples given it is clear that they are very rare. Why the
variant with the inclusion of this functionality into the math module was
not considered at all. If the answer is - consistency upon the numeric
tower - will it go for complex type and what will it mean (there can be two
point of views)?

Is this functionality so often used and practical to be a method of float,
int, ..., and not just to be an auxiliary function?

p.s.: The same thoughts about `as_integer_ratio` discussion.

With kind regards,
-gdg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do we have vlookup function

[Stephane Wirtel via Python-Dev]

This mailing list is for the development of CPython, not for the end
user, please could you move your question on python-l...@python.org ?

Thank you,

Le 22/03/18 à 07:27, Rohit Adhikari a écrit :
> Do we have vlookup function which can be used in dataframe same as used
> in excel.
> Can you please provide the pointer for the same?
> 
> Thanks!!!
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Do we have vlookup function

[Rohit Adhikari]

Do we have vlookup function which can be used in dataframe same as used in
excel.
Can you please provide the pointer for the same?

Thanks!!!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Tim Peters]

[Tim]
>> from trig functions doing argument reduction as if pi were represented
>> with infinite precision,

[Greg Ewing ]
> That sounds like an interesting trick! Can you provide
> pointers to any literature describing how it's done?
>
> Not doubting it's possible, just curious.

As I recall, when it was first done a "lazy" routine produced as many
bits of pi as a given argument required, doing gonzo arbitrary
precision arithmetic.

Later, computer-aided analysis based on continued fraction expansions
identified the worst possible case across all IEEE doubles (&
singles).  For example, it's possible in reasonable time to find the
IEEE double that comes closest to being an exact integer multiple of
pi/4 (or whatever other range you want to reduce to).  Then it's only
necessary to precompute pi to as many bits as needed to handle the
worst case.

In practice, falling back to that is necessary only for "large"
arguments, and the usual double-precision numeric tricks suffice for
smaller arguments.

Search the web for "trig argument reduction" for whatever the state of
the art may be today ;-)

For actual code, FDLIBM does "as if infinite precision" trig argument
reduction, using a precomputed number of pi bits sufficient to handle
the worst possible IEEE double case, and is available for free from
NETLIB:

http://www.netlib.org/fdlibm/

The code is likely to be baffling, though, as there's scant
explanation.  Reading a paper or two first would be a huge help.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecating float.is_integer()

[Greg Ewing]

Tim Peters wrote:

from trig functions doing argument reduction as if pi were represented
with infinite precision,


That sounds like an interesting trick! Can you provide
pointers to any literature describing how it's done?

Not doubting it's possible, just curious.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com