Re: What is your experience porting Python 2.7.x scripts to Python 3.x?

[Robin Becker]

On 22/01/2019 19:00, Schachner, Joseph wrote:
..

For anyone who has moved a substantial bunch of Python 2 to Python 3,   can you 
please reply with your experience?  Did you run into any issues?   Did 2to3 do 
its job well, or did you have to review its output to eliminate some working 
but silly thing?


,..
I did the port of reportlab (www.reportlab.com) from  code supporting 2.x only x>=3 to a version which supported 2.7.z & >=3.3. 
The reportlab toolkit was then about 14 years old and had (and still has lots of cruft). The port effort began 20130215 and ended 
20140326 ie 13 months. There were 333 commits on the branch. I used 2to3, but not six. Because we needed to maintain 2.7 and >=3.3 
there were quite a few issues related to simple things like iterkeys/values/items <--> keys/values/items removal of xrange etc etc.


Maintaining compatible extensions is also hard. Pdf production requires byte output and was already quite messy because reportlab 
supported both utf8 and unicode inputs; it hasn't got a lot easier.


As for performance the 3.x runs were generally slower than 2.7, but I think that situation has changed with 3.6 & 3.7 (for the 
reportlab tests on windows 2.7 takes 68.7", 3.4 83.8", 3.5 77.0", 3.6 61.5" & 3.7 60.9").


At some point reportlab will be made 3.x only which will require more effort.
--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list

ANN: A new version (0.4.4) of python-gnupg has been released. It contains a security-related change - please update to this version

[Vinay Sajip via Python-list]

A new version of the Python module which wraps GnuPG has been released.
What Changed?=This is an enhancement and security-fix release, and 
all users are stronglyencouraged to upgrade.
Brief summary:
* Fixed #108: Changed how any return value from the on_data callable is  
processed. In earlier versions, the return value was ignored. In this version,  
if the return value is False, the data received from gpg is not  buffered. 
Otherwise (if the value is None or True, for example), the  data is buffered as 
normal. This functionality can be used to do your own  buffering, or to prevent 
buffering altogether.
  The on_data callable is also called once with an empty byte-string to  signal 
the end of data from gpg.
* Fixed #97: Added an additional attribute check_fingerprint_collisions to  GPG 
instances, which defaults to False. It seems that gpg is happy  to have 
duplicate keys and fingerprints in a keyring, so we can't be too  strict. A 
user can set this attribute of an instance to True to trigger a  check for 
collisions.
* Fixed #111: With GnuPG 2.2.7 or later, provide the fingerprint of a signing  
key for a failed signature verification, if available.
* Fixed #21: For verification where multiple signatures are involved, a  
mapping of signature_ids to fingerprint, keyid, username, creation date,  
creation timestamp and expiry timestamp is provided.
* Added a check to disallow certain control characters ('\r', '\n', NUL) in  
passphrases. This fix mitigates against CVE-2019-6690.
This release [2] has been signed with my code signing key:
Vinay Sajip (CODE SIGNING KEY) Fingerprint: CA74 
9061 914E AC13 8E66 EADB 9147 B477 339A 9B86
Recent changes to PyPI don't show the GPG signature with the download links.An 
alternative download source where the signatures are available is the 
project'sown downloads page [5].
What Does It Do?The gnupg module allows Python programs to make 
use of thefunctionality provided by the Gnu Privacy Guard (abbreviated GPG 
orGnuPG). Using this module, Python programs can encrypt and decryptdata, 
digitally sign documents and verify digital signatures, manage(generate, list 
and delete) encryption keys, using proven Public KeyInfrastructure (PKI) 
encryption technology based on OpenPGP.
This module is expected to be used with Python versions >= 2.4, as itmakes use 
of the subprocess module which appeared in that version ofPython. This module 
is a newer version derived from earlier work byAndrew Kuchling, Richard Jones 
and Steve Traugott.
A test suite using unittest is included with the source distribution.
Simple usage:
>>> import gnupg>>> gpg = gnupg.GPG(gnupghome='/path/to/keyring/directory')>>> 
>>> gpg.list_keys()
[{...'fingerprint': 'F819EE7705497D73E3CCEE65197D5DAC68F1AAB2','keyid': 
'197D5DAC68F1AAB2','length': '1024','type': 'pub','uids': ['', 'Gary Gross (A 
test user) ']},{...'fingerprint': 
'37F24DD4B918CC264D4F31D60C5FEFA7A921FC4A','keyid': 
'0C5FEFA7A921FC4A','length': '1024',...'uids': ['', 'Danny Davis (A test user) 
']}]>>> encrypted = gpg.encrypt("Hello, world!", 
['0C5FEFA7A921FC4A'])>>> str(encrypted)
'-BEGIN PGP MESSAGE-\nVersion: GnuPG v1.4.9 
(GNU/Linux)\n\nhQIOA/6NHMDTXUwcEAf.-END PGP MESSAGE-\n'>>> decrypted = 
gpg.decrypt(str(encrypted), passphrase='secret')>>> str(decrypted)
'Hello, world!'>>> signed = gpg.sign("Goodbye, world!", passphrase='secret')>>> 
verified = gpg.verify(str(signed))>>> print "Verified" if verified else "Not 
verified"
'Verified'
As always, your feedback is most welcome (especially bug reports [3],patches 
and suggestions for improvement, or any other points via themailing 
list/discussion group [4]).
Enjoy!
Cheers
Vinay SajipRed Dove Consultants Ltd.
[1] https://bitbucket.org/vinay.sajip/python-gnupg[2] 
https://pypi.python.org/pypi/python-gnupg/0.4.4[3] 
https://bitbucket.org/vinay.sajip/python-gnupg/issues[4] 
https://groups.google.com/forum/#!forum/python-gnupg[5] 
https://bitbucket.org/vinay.sajip/python-gnupg/downloads/
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: ValueError: Input contains NaN, infinity or a value too large for dtype('float32')

[Rhodri James]

On 23/01/2019 22:29, paulmatth...@gmail.com wrote:

You may be using the sklearn package incorrectly; you'll
have to read the (apparently quite prolific) documentation yourself,
I've never used it.

--
Rhodri James *-* Kynesim Ltd


So why would you try to answer it?  I have the same issue but there are no NaN, 
infinites, negative infinites or large values.



I tried to answer it probably because it looked fishy and some people 
need encouragement to read the fine manual.  I don't know for sure, you 
trimmed all the attribution and I can't find a thread of this name in 
the archives for last year.  What is "the same issue"?


--
Rhodri James *-* Kynesim Ltd
--
https://mail.python.org/mailman/listinfo/python-list

Re: What is your experience porting Python 2.7.x scripts to Python 3.x?

[Robin Becker]

On 23/01/2019 21:51, Ian Kelly wrote:

On Wed, Jan 23, 2019 at 1:36 PM Stefan Behnel  wrote:



.

All right, but apart from absolute imports, the print function, and true
division, what has Python 3.x ever done for us?

*ducks*


headaches :)

--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list

Re: What is your experience porting Python 2.7.x scripts to Python 3.x?

[Pete Forman]

Robin Becker  writes:

> On 22/01/2019 19:00, Schachner, Joseph wrote:
> ..
>> For anyone who has moved a substantial bunch of Python 2 to Python 3,
>> can you please reply with your experience? Did you run into any
>> issues? Did 2to3 do its job well, or did you have to review its
>> output to eliminate some working but silly thing?
>>
> ,..
> I did the port of reportlab (www.reportlab.com) from code supporting
> 2.x only x>=3 to a version which supported 2.7.z & >=3.3. The
> reportlab toolkit was then about 14 years old and had (and still has
> lots of cruft). The port effort began 20130215 and ended 20140326 ie
> 13 months. There were 333 commits on the branch. I used 2to3, but not
> six. Because we needed to maintain 2.7 and >=3.3 there were quite a
> few issues related to simple things like iterkeys/values/items <-->
> keys/values/items removal of xrange etc etc.
>
> Maintaining compatible extensions is also hard. Pdf production
> requires byte output and was already quite messy because reportlab
> supported both utf8 and unicode inputs; it hasn't got a lot easier.
>
> As for performance the 3.x runs were generally slower than 2.7, but I
> think that situation has changed with 3.6 & 3.7 (for the reportlab
> tests on windows 2.7 takes 68.7", 3.4 83.8", 3.5 77.0", 3.6 61.5" &
> 3.7 60.9").
>
> At some point reportlab will be made 3.x only which will require more
> effort.

Packages like reportlab with a need to support both Python 2 and 3 end
up with the worst of both worlds. The initial drive for Py3k was to drop
cruft that had accumulated over the years. Mixing old and new hampers
your ability to write clean 3 code.

-- 
Pete Forman
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How to force the path of a lib ?

[Vincent Vande Vyvre]

Le 23/01/19 à 13:11, Neal Becker a écrit :

dieter wrote:


Vincent Vande Vyvre  writes:
.




To load external C/C++ shared objects, the dynamic lickage loader
(ldd) is used. "ldd" does not look at Pthon's "sys.path".
Unless configured differently, it looks at standard places
(such as "/usr/lib/x86_64-linux-gnu").

You have several options to tell "ldd" where to look for
shared objects:

  * use the envvar "LD_LIBRARY_PATH"
This is a "path variable" similar to the shell's "PATH",
telling the dynamic loader in which directories (before
the standard ones) to look for shared objects

  * use special linker options (when you link your Python
extension shared object) to tell where dependent shared
object can be found.


To follow up on that last point, look up --rpath and related.

Trying the first solution with "export 
LD_LIBRARY_PATH=/home/vincent/CPython/py370_venv/lib" has no effect.


The second solution is better, after looking de doc of 
distutils.core.Extension I've added this line into my setup.py

    Extension('libexiv2python',
    ...,
runtime_library_dirs=['/home/vincent/CPython/py370_venv/lib/'],
    ...

rebuild, install and test
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyexiv2
>>>

Great!

Many thanks,
Vincent

--
https://mail.python.org/mailman/listinfo/python-list

preferences file

[Dave]

I'm doing a small application and want to add user preferences.  Did 
some googling to see if there are standard Python ways/tools, but it 
seems not so much.  My specific questions are:


1. Best practices for a user preference file/system?

2. File format favored and why - ini, JSON, etc?

3. File location?  I'm using Ubuntu and I believe that the correct 
location would be home/.config/ .  What about Mac and Windows?


Would like to find a Python library that handles all of this, but so far...

Thanks,
Dave
--
https://mail.python.org/mailman/listinfo/python-list

Re: What is your experience porting Python 2.7.x scripts to Python 3.x?

[Tim Chase]

On 2019-01-22 19:20, Grant Edwards wrote:
> > For anyone who has moved a substantial bunch of Python 2 to Python
> > 3, can you please reply with your experience?  
> 
> If you used bytes (or raw binary strings) at all (e.g. for doing
> things like network or serial protocols) you're in for a lot of
> pain.

This was my biggest pain point, but it was a good pain.  At $DAYJOB we
had files coming from customers and telecom providers where the
encoding had never been specified.  By going through the conversion
process, we were able to formalize the encoding of various files
meaning fewer crashes when some unexpected character would slip in
and fail to convert.

A painful process, but the end result was better in a multitude of
ways.

-tkc


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: checking protocols.

[DL Neil]

Avi


Haven't noticed an answer to this. Did I miss anything?

I never saw the original message appear and thus expected no replies. I see a 
later post by Chris indicating he did not see it either. I assumed perhaps a 
moderator needed to approve it.


=a silly question on my part. A quick check of the list archive shows 
your original post but no previous responses.




My question really boils down to being intrigued how the python language can be 
extended using documented protocols in more ways and having a place where you 
can see a list of the ones out there and a way to join in with your own classes 
reliably and so on.


=if I am interpreting your intent correctly (equally that of Pythonic 
thinking) the idea is for 'more' to happen at compile-time. This 
expectation likely comes?came from our earlier training in more 
strict/rigid languages. Whereas many characteristics of Python flow 
directly as consequences of it being (chosen/designed to be) a dynamic 
language - and thus many?most such decisions delayed until run-time.


=The first (related) example which springs to mind is object 
polymorphism. It must be so much harder to achieve such incredible power 
and flexibility (as Python does) when 'rules' are 'cast in stone' early.


=However, it might be worth making suggestions generated 'here' to the 
(hard-working) docs team. There are discussions of specific protocols, 
in their individual locations throughout the manual/library. Is it 
sufficiently clear what is expected when building our own classes 
according to 'the' protocol?


=Rather than collecting all the protocols' documentation together (cf 
the sequence currently used), maybe an cross-index somewhere?




I note there may be many unpublished protocols used internally within modules 
that also create similar structures but do not expect to be emulated.


=coming from Python, the PSL, or third-parties?



I see interpolated comments below and will interpolate within that but it gets 
confusing.


=am using line-prefix to help differentiate. Interleaved comments help 
(?me) with following the conversational flow - and ensuring that each 
point/question is addressed (he hopes)




On 20/01/19 11:07 AM, Avi Gross wrote:

Short question. Checking if a protocol is set up?


=do you mean that to check/require that a class exhibits a particular protocol 
we should use abstract classes - will not instantiate unless all the required 
components are coded?

== I am not looking for any specific options but indeed including an 
abstract class might be one way to go. It might supply a reasonable way to ask 
if the class you make yourself INTENDS on running that protocol and would 
enforce the presence of the required methods somewhere in the inheritance 
chain. Of course the functions could be stubs that don't do the right thing, or 
anything passable.

... (previous content removed, as no longer necessary)


=The phrase "we're all adults here" is often used to justify Python's 
apparent inexactitude (cf other languages*). However, I often wonder 
whether that is merely an excuse to avoid explaining oneself properly. 
For example, 'here' (or is it another list, strictly speaking?) I read 
ANNouncements of package updates. How many tell me wonderful additions 
to the package, how much work went into them, what I'd now be able to do 
with it, and who I should credit for all such goodness - but don't 
actually get around to mentioning the package's objectives or 
applicability? Unless I know said package, these ANNs are a waste of my 
reading-time!


=On the other hand, such 'openness' as non-enforced "private variables" 
(applying the 'adults' phrase) enables us to consider them "private" in 
the normal course, but to abuse the notion should the need take us (and 
caution have been thrown to the wind)!


=Back to protocols: and if I want to code certain magic methods, etc, 
comprising a protocol, should I be allowed to choose to ignore other(s)? 
Be it on my own head!


* comparing Python to other languages is (IMHO) neither a reliable nor 
particularly logical argument. How can we be 'the best' if all we do is 
copy and reproduce everyone else - and would that even be possible? 
Surely the valid comparison is between the problems which must be solved 
and the facilities of the language chosen to code the solution?


=That said, there is no question that Python has been developing 
somewhat schizoid tendencies. Another mantra has always been: 'it is 
better to ask forgiveness than permission'*, and thus we should wrap 
assumptive code in a try...except block. This change took a long time to 
sit comfortably in my (albeit small) brain - and possibly still requires 
conscious consideration. My thought pattern** is still: first check the 
data, then ..., ie ensure denominator is not zero before dividing rather 
than


try:
divide
except (whatever):
print( "Not by zero, bub!" )

* (originally attributed to Adm Grace Hopper of COBOL fa

Re: preferences file

[Chris Warrick]

On Thu, 24 Jan 2019 at 20:50, Dave  wrote:
>
> I'm doing a small application and want to add user preferences.  Did
> some googling to see if there are standard Python ways/tools, but it
> seems not so much.  My specific questions are:
>
> 1. Best practices for a user preference file/system?

Put them in the appropriate config directory (see answer to Q3).
Windows has the registry and macOS has User Defaults, but a custom
format is fine, especially if you’re making something multi-platform.

> 2. File format favored and why - ini, JSON, etc?

If you want/expect humans to edit it, go with INI (configparser).
Otherwise (if the editing is done via your app with a nice UI), JSON
can express more than just key-value mappings, although there are no
comments, and its strict syntax requirements can be problematic for
non-programmers trying to edit the file. Randomly loading .py or
pickle files can potentially be unsafe; writing .py files in an
automated way is not trivial. I wouldn’t bother with any other
formats, mainly since they are not supported in stdlib, although if
you need human editing and more advanced structures, then maybe
YAML/TOML (they aren’t as foolproof as INI though).

> 3. File location?  I'm using Ubuntu and I believe that the correct
> location would be home/.config/ .  What about Mac and Windows?

https://pypi.org/project/appdirs/

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: preferences file

[DL Neil]

Dave,

On 25/01/19 8:42 AM, Dave wrote:
I'm doing a small application and want to add user preferences.  Did 
some googling to see if there are standard Python ways/tools, but it 
seems not so much.  My specific questions are:


1. Best practices for a user preference file/system?

> [edited]
> Would like to find a Python library that handles all of this, but so 
far...


=Had a similar need. Also looked around, but failed. Will be interested 
in any discussion that follows...




2. File format favored and why - ini, JSON, etc?


=Am no great fan of .ini (a) where it is most used, ie cynicism and 
bias; and (b) and because it is too "flat".


=Initially thought to use .conf, comforted by frequent use on Linux 
servers, but similar criticism.


(at this point the PSL's configparser bows out)

=Looked at .JSON and liked the syntactic similarities to Python.

(utility available in PSL)

=Settled on .YAML, mostly because not very different to .JSON yet has 
specific, applicable design philosophy.


(IIRC picked this up through pip3)

=Ended-up coding my own little utility to call from any/every 
application; based on import yaml. Will be interested to see if someone 
suggests something I missed back-then...



=Have been gradually adding further facilities, eg commandLN overrides. 
Also, been wondering about its applicability to my web work - 
particularly when it comes to separating versions, eg "live" from (user 
acceptance) "test"...


=Once done, I've noticed myself coding fewer global constants, instead 
putting them into the .yaml configuration file. Of course, this adds a 
point-of-failure - particularly if the user has access/editing rights on 
the file (but with great power, comes ...!)



3. File location?  I'm using Ubuntu and I believe that the correct 
location would be home/.config/ .  What about Mac and Windows?


=Makes good sense, but ultimately depends upon usage/the application/how 
it is to be delivered to the users (and possibly more). Obviously if the 
user is not on that machine, or you run on behalf of several users that 
won't work (perhaps implement separate .../userNM/ sub-directories and 
use such for both config input and any output files/reports).


=Again, will be interested to see others' ideas...

(this smart-alec locates his windows in the wall, and keeps his mac in 
the cupboard/closet)


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: preferences file

[eryk sun]

On 1/24/19, Dennis Lee Bieber  wrote:
> On Thu, 24 Jan 2019 14:42:59 -0500, Dave  declaimed
> the following:
>>
>>3. File location?  I'm using Ubuntu and I believe that the correct
>>location would be home/.config/ .  What about Mac and Windows?
>>
>   Windows?
>
>   %UserProfile%\Roaming\

That should be "%UserProfile%\AppData\Roaming". This is the default
path, which should never be used directly because "Roaming" is
relocatable. Use "%AppData%" instead.

That said, relying on environment variables isn't the best practice.
The value might be wrong if the folder was relocated after our process
tree was created. Use the shell's well-known-folder interface instead.
For example:

import pythoncom
from win32com.shell import shell

kfmgr = pythoncom.CoCreateInstance(
shell.CLSID_KnownFolderManager, None,
pythoncom.CLSCTX_INPROC_SERVER,
shell.IID_IKnownFolderManager)

appdata = kfmgr.GetFolder(shell.FOLDERID_RoamingAppData)
appdata_path = appdata.GetPath()

IKnownFolderManager interface:
https://docs.microsoft.com/en-us/windows/desktop/api/shobjidl_core/nn-shobjidl_core-iknownfoldermanager

IKnownFolder interface:
https://docs.microsoft.com/en-us/windows/desktop/api/shobjidl_core/nn-shobjidl_core-iknownfolder
-- 
https://mail.python.org/mailman/listinfo/python-list