Re: My environment doesn't load

[Fabien LUCE]

Well, Beautifulsoup is not 
On Wed, Oct 10, 2018 at 02:installed on my system, just in the
environment, so how could apache run correctly my app without accessing my 
environment?
I thought that venv or virtualenv point was only to create some folder
which was simply sourced. What other things they do?


34:10AM +1100, Chris Angelico wrote:
> On Wed, Oct 10, 2018 at 2:32 AM  wrote:
> >
> > I rsynced the full local project directory (which contains the virtualenv 
> > dir) to a remote server.
> > And there I couldn't enter into it.
> > However, obviously apache2 achieved to...
> 
> Not so obvious. It's possible Apache is ignoring the venv and just
> running the app in its own environment.
> 
> As others have said, you should maintain a requirements.txt and then
> create a virtual environment on the target system. Since you're on a
> recent Python, ignore the old "virtualenv" package, and use "venv"
> from the stdlib.
> 
> ChrisA
> -- 
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: My environment doesn't load

[Fabien LUCE]

I rsynced the full local project directory (which contains the virtualenv dir) 
to a remote server.
And there I couldn't enter into it.
However, obviously apache2 achieved to...


October 9 2018 5:21 PM, "Chris Warrick"  wrote:
> On Tue, 9 Oct 2018 at 11:18,  wrote:
> 
>> Here are the ouput during sourcing:
>> 
>>> [ftg @ localhost] [/var/www/ffablob]
>>> % source env-p3/bin/activate
>>> [ftg @ localhost] [/var/www/ffablob]
>>> % which python
>>> /usr/bin/python
>> 
>> and if I run some of my code that import beautyfulsoup it fails (python 
>> mycode.py), althoug running
>> directly ./env-p3/python3.5 mycode.py is working...
> 
> Was the virtualenv copied between machines or directories? If yes: you
> can’t do that, you must create a new virtualenv in the desired
> location and install dependencies to it.
> 
> --
> Chris Warrick 
> PGP: 5EAAEA16
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: My environment doesn't load

[Fabien LUCE]

Here are the ouput during sourcing:

[ftg @ localhost] [/var/www/ffablob]
% source env-p3/bin/activate

   
[ftg @ localhost] [/var/www/ffablob]
% which python  

   
/usr/bin/python

and if I run some of my code that import beautyfulsoup it fails (python 
mycode.py), althoug running directly ./env-p3/python3.5 mycode.py is working...








October 9 2018 11:06 AM, "Thomas Jollans"  wrote:
> On 2018-10-09 10:36, f...@lutix.org wrote:
> 
>> Thx,
>> 
>> initially the env was created with virtualenv yes, and then I copied the 
>> folder of the project on a
>> remote server. I am using zsh (both locally and remotely).
>> My PATH env variable does includes well the path of my env bin. In this 
>> folder there are symbolic
>> links.
>> And yes there is well an /env/bin/python in there:
> 
> But can you run it?
> 
> If it exists, and it's at the top of your PATH, then everything should
> be fine unless your shell's command hash table is getting in the way -
> which it shouldn't. virtualenv knows about those:
> 
> % tail -6 bin/activate
> # This should detect bash and zsh, which have a hash command that must
> # be called to get it to forget past commands. Without forgetting
> # past commands the $PATH changes we made may not be respected
> if [ -n "$BASH" -o -n "$ZSH_VERSION" ] ; then
> hash -r 2>/dev/null
> fi
> 
>> drwxrwxr-x 3 ftg ftg 4096 Jul 6 21:38 .
>> drwxrwxr-x 6 ftg ftg 4096 May 29 20:33 ..
>> -rw-rw-r-- 1 ftg ftg 2114 May 29 20:30 activate
>> -rw-rw-r-- 1 ftg ftg 1056 May 29 20:30 activate.csh
>> -rw-rw-r-- 1 ftg ftg 2210 May 29 20:30 activate.fish
>> -rw-rw-r-- 1 ftg ftg 1137 May 29 20:30 activate_this.py
>> -rwxrwxr-x 1 ftg ftg 276 Jul 6 21:38 chardetect
>> -rwxrwxr-x 1 ftg ftg 318 May 29 20:33 django-admin
>> -rwxrwxr-x 1 ftg ftg 176 May 29 20:33 django-admin.py
>> -rwxrwxr-x 1 ftg ftg 9267 Jul 6 21:38 dumppdf.py
>> -rwxrwxr-x 1 ftg ftg 285 May 29 20:30 easy_install
>> -rwxrwxr-x 1 ftg ftg 285 May 29 20:30 easy_install-3.5
>> -rwxrwxr-x 1 ftg ftg 2438 Jul 6 21:38 latin2ascii.py
>> -rwxrwxr-x 1 ftg ftg 5970 Jul 6 21:38 pdf2txt.py
>> -rwxrwxr-x 1 ftg ftg 267 May 29 20:30 pip
>> -rwxrwxr-x 1 ftg ftg 267 May 29 20:30 pip3
>> -rwxrwxr-x 1 ftg ftg 267 May 29 20:30 pip3.5
>> drwxrwxr-x 2 ftg ftg 4096 Jul 6 21:38 __pycache__
>> lrwxrwxrwx 1 ftg ftg 7 May 29 20:30 python -> python3
>> -rwxrwxr-x 1 ftg ftg 4743656 May 29 20:30 python3
>> lrwxrwxrwx 1 ftg ftg 7 May 29 20:30 python3.5 -> python3
>> -rwxrwxr-x 1 ftg ftg 2373 May 29 20:30 python-config
>> -rwxrwxr-x 1 ftg ftg 264 May 29 20:30 wheel
>> 
>> So I guess that copying an environment is not "creating" it remotely.
>> However it works in a certain point of view because apache can find it an 
>> use it as python-path is
>> set for my WSGI Daemon Process.
>> DOn't understand...
>> 
>> October 9 2018 10:18 AM, "Thomas Jollans"  wrote:
>>> On 09/10/2018 09:20, f...@lutix.org wrote:
>>> 
 Hello,
> I have rsynced a folder on my distant server. Now I try to source my 
> environment: source
 
 ./env/bin/activate and nothing happens.
 When type > which python, /usr/bin/python is still printed. What could be 
 the issue?
> Thanks
>>> 
>>> What shell are you using? Was the environment created by virtualenv, venv 
>>> or something else? After
>>> sourcing activate, what is your PATH? Is there a ./env/bin/python? Does it 
>>> work?
>>> 
>>> (The hopefully easy fix is obviously to just recreate the environment 
>>> locally from your
>>> requirements.txt, assuming you have one)
>>> -- https://mail.python.org/mailman/listinfo/python-list
> 
> --
> Thomas Jollans
> 
> m ☎ +31 (6) 42630259
> e ✉ t...@tjol.eu
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Design of my project

[Fabien LUCE]

 Hello everyone,

Here is a small picture of my project.
I'd like to fetch several datas from a website representing races results.
Each line of the result contains rank, runner's name and other attributes
of the runner (like club etc...).
For now I have created 2 classes (Race and Runner).
Methods of Race allow me to fetch datas on the www thanks to beautifulsoup
module.I instanciate a race at the beginning of the scan and for each line
I instanciate a Runner.
I want to store all those information in a database (for both Race and
Runner), but here is the point, I have the feeling, on a programming
elegance point of view, that I have to separate methods that fetch from
method that store and also have a kind of superior object that controls
those 2 features.
Moreover fetching will interact with storage: if race is already in
database, no need to fetch again.
How should I "design" this? Should I have only (despite my first
impression) one class and all the methods in it? Should I create a parent
class along with 2 childrens that manage the two kind of methods?
Is there a design pattern for this?

I precise I am relatively new in "intelligent" python programming ;)

Thanks
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: String encoding in Py2.7

[Fabien LUCE]

May 29 2018 11:12 AM, "Thomas Jollans"  wrote:
> On 2018-05-29 09:55, f...@lutix.org wrote:
> 
>> Hello,
>> Using Python 2.7 (will switch to Py3 soon but Before I'd like to understand 
>> how string encoding
>> worked)
> 
> Oh dear. This is probably the exact wrong way to go about it: the
> interplay between string encoding, unicode and bytes is much less clear
> and easy to understand in Python 2.

Ok I will quickly jump into py3 then.

> 
>> Could you please tell me is I understood well what occurs in Python's mind:
>> in a .py file:
>> if I write s="héhéhé", if my file is declared as unicode coding, python will 
>> store in memory
>> s='hx82hx82hx82'
> 
> No, it doesn't. At the very least, you're missing some backslashes – and
> I don't know of any character encoding that using 0x82 to encode é.
> 
 surprinsingly backslash were removed from my initial text...
ok so stored raw bytes are the one processed by the system encoder. If my 
console were utf-8 I would have same raw bytes string than you. 
  

> On my system, I see
> 
 s = 'héhéhé'
 s
> 
> 'h\xc3\xa9h\xc3\xa9h\xc3\xa9'
> 
> My system uses UTF-8. If your PC is set up to uses an encoding like ISO
> 8859-15 or Windows-1252, you should see
> 
> 'h\xe9h\xe9h\xe9'
> 
> The \x?? are just Python notation.
> 
>> however this is not yet unicode for python interpreter this is just raw 
>> bytes. Right?
> 
> Right, this is a bunch of bytes:
> 
 s
> 
> 'h\xe9h\xe9h\xe9'
> 
 [ord(c) for c in s]
> 
> [104, 233, 104, 233, 104, 233]
> 
 [hex(ord(c)) for c in s]
> 
> ['0x68', '0xe9', '0x68', '0xe9', '0x68', '0xe9']
> 
 
>> 
>> By the way, why 'h' is not turned into hexa value? Because it is already in 
>> the ASCII table?
> 
> That's just how Python 2 likes to display stuff.
> 
>> If I want python interpreter to recognize my string as unicode I have to 
>> declare it as unicode
>> s=u'héhéhé' and magically python will look for those
>> hex values 'x82' in the Unicode table. Still OK?
> 
> In principle, the unicode table has nothing to do with anything here. It
> so happens that for some characters in some encodings the value is equal
> to the code point, but that's neither here nor there.
> 
>> Now: how come when I declare s='héhéhé', print(s) displays well 'héhéhé'? Is 
>> it because of my shell
>> windows that is dealing well with unicode? Or is it
>> because the print function is magic?
> 
> It's because the print statement is magic.
> 
> Actually, this *only* works if the encoding of your file matches the
> default encoding required by your console. This is usually the case as
> long as you stay on the same PC, but this assumption can fall apart
> quite easily when you move code and data between systems, especially if
> they use different operating systems or (human) languages.
> 
> Just use Python 3. There, the print function is not magic, which makes
> life so much more logical.

Thanks
> 
> -- Thomas
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Let's talk about debuggers!

[Fabien]

On 10/25/2017 03:07 PM, Thomas Jollans wrote:

What options are there for Python (that work)?


PyCharm's debugger is fine (also available in the community edition)

Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Easier way to do this?

[Fabien]

On 10/05/2017 12:00 AM, Thomas Jollans wrote:

On 04/10/17 22:47, Fabien wrote:

On 10/04/2017 10:11 PM, Thomas Jollans wrote:

Be warned, pandas is part of the scientific python stack, which is
immensely powerful and popular, but it does have a distinctive style
that may appear cryptic if you're used to the way the rest of the world
writes Python.

Can you elaborate on this one? As a scientist, I am curious;-)

Sure.

Python is GREAT at iterating. Generators are everywhere. Everyone loves
for loops. List comprehensions and generator expressions are star
features. filter and map are builtins. reduce used be a builtin, even
though almost nobody really understood what it did.

In [1]: import numpy as np

In the world of numpy (and the greater scientific stack), you don't
iterate. You don't write for loops.


Right, now I see what you meant. The first thing I teach to my students 
is that loops are evil, and that they should always avoid them when 
possible ;-)


Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Easier way to do this?

[Fabien]

On 10/04/2017 10:11 PM, Thomas Jollans wrote:

Be warned, pandas is part of the scientific python stack, which is
immensely powerful and popular, but it does have a distinctive style
that may appear cryptic if you're used to the way the rest of the world
writes Python.


Can you elaborate on this one? As a scientist, I am curious ;-)
--
https://mail.python.org/mailman/listinfo/python-list

Re: PYTHON GDAL

[Fabien]

On 07/14/2017 03:57 PM, jorge.conr...@cptec.inpe.br wrote:



Hi,

I installed the GDAL 2.2.1 using conda. Then I did:

import gdal

and I had:


Traceback (most recent call last):
   File "", line 1, in 
   File "/home/conrado/miniconda2/lib/python2.7/site-packages/gdal.py", 
line 2, in 

 from osgeo.gdal import deprecation_warn
   File 
"/home/conrado/miniconda2/lib/python2.7/site-packages/osgeo/__init__.py", line 
21, in 

 _gdal = swig_import_helper()
   File 
"/home/conrado/miniconda2/lib/python2.7/site-packages/osgeo/__init__.py", line 
17, in swig_import_helper

 _mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: libicui18n.so.56: cannot open shared object file: No such 
file or directory



then I used the command find:

find . -name 'libicui18n.so.56' -print

and I had:

./usr/local/lib/python3.6/site-packages/PyQt5/Qt/lib/libicui18n.so.56


Please, what can I do to put (set) this lib for python2.7 recognize it.



Since you are using conda I *strongly* recommend to use the conda-forge 
channel to install GDAL:


conda install -c conda-forge gdal

https://conda-forge.org/





Thanks,


Conrado


--
https://mail.python.org/mailman/listinfo/python-list

Re: [GIS] Keeping only POIs X kms from a trace?

[Fabien]

On 05/08/2017 02:34 PM, Heck Lennon wrote:

To plan bike tours, I'd like to…
1) import a trace from a GPX file,
2) import Archie's list for a given country,
3) filter that list to only keep campsites that are 5 kms from the route, and
4) export the output to a GPX file.

Since a "gpxpy" Python module is available, I assume this can be done just with 
a script instead of an expert application like QGIS… but I know nothing about GIS.

Can the above be done with a Python script?


Yes, you could also have a look at the excellent libraries:
- Geopandas: http://geopandas.org/
- Shapely: http://toblerity.org/shapely/manual.html

Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

[issue29635] os.chdir() acts like `cd -P` by default, it should offer an option to not follow symlinks

[Fabien Dubosson]

Fabien Dubosson added the comment:

> I'm going to close this issue. 

I was doing to do so, here was my message:



> See shells sources. According to the manpage `pwd -L` just uses PWD from 
> environment.

I looked directly at `pwd` sources, and indeed it is using $PWD [1].

[1] 
http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/pwd.c?id=509152bdd47a278dc00ddaf3200ac65044b836b1#n305

So it looks like `cd -L` is just some bash internal dealing with $PWD. Probably 
not something useful enough to be in the standard library, closing it then.

Sorry for the incovenience

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29635>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29635] os.chdir() acts like `cd -P` by default, it should offer an option to not follow symlinks

[Fabien Dubosson]

Fabien Dubosson added the comment:

> Most os functions are thin wrappers around system calls. There is no system 
> call that works as 'cd' without '-P'.

I would like to believe in this, but then if `cd` is some bash internal, how 
does `/usr/bin/pwd -L` find it back?

> Since yours is the itch, I'm afraid you are going to have to be the one to 
> figure out how this could be implemented :)

Once I'll have figured out how all this is working, maybe. Not sure it would me 
in my competencies though.

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29635>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29635] os.chdir() acts like `cd -P` by default, it should offer an option to not follow symlinks

[Fabien Dubosson]

Fabien Dubosson added the comment:

Thanks for the additional information!

> The 'cd' man page says that 'cd -P' should "perform actions equivalent to the 
> chdir() function".

Just wondering, do you know what is the function called by `cd`/`cd -L` then? 
It doesn't seems to be a bash internal tweak because `/usr/bin/pwd -L` is able 
to get the symlink path, and this binary is not part of bash.

> You'll have to make a case for it being useful enough to add.

The use case is for shells written in Python, like for instance `xonsh` [1,2]. 
Being powered by Python, the only way to change directory is by using 
`os.chdir()`. It is then not possible to mimic the bash `cd` function without 
using workarounds, like for instance storing the symlink path in a variable, 
and even this doesn't allow to use `/usr/bin/pwd -L` in scripts because it 
would always returns the physical location.

Having such described functions (yet to be named) would permit python shells to 
offer users differentiated `cd` and `cd -P` commands, as well as having `pwd 
-L` in scripts behaving the same than in traditional shells.

[1] http://xon.sh/
[2] https://github.com/xonsh/xonsh

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29635>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29635] os.chdir() acts like `cd -P` by default, it should offer an option to not follow symlinks

[Fabien Dubosson]

Fabien Dubosson added the comment:

The previous example (sorry, first time using this platform, I'm a little bit 
messy here), shows that `os.chdir` resolves symlinks by default, as opposed to 
what `cd` does in bash. This means it is not possible to change the directory 
to a symlink folder.

A solution would be a keyword argument in `os.chdir` and `os.getcwd` that would 
mimic the `-P` and `-L` arguments of `cd` and `pwd`, like:

os.chdir(logical=False)  # or follow_symlinks=True maybe
os.getcwd(logical=False)

I don't know what should be the default values for these args, but at least 
having an option to change the behavior would be nice.

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29635>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29635] os.chdir() acts like `cd -P` by default, it should offer an option to not follow symlinks

[Fabien Dubosson]

New submission from Fabien Dubosson:

When using bash, the `cd` function does not follow symlinks by default, but `cd 
-P` does. The `os.chdir` function behaves like `cd -P` preventing to be able to 
change directory to a symlink folder.

Initial setup (make a `/tmp/to/dst` symlink pointing to `/tmp/from/src`):

cd /tmp
mkdir from from/src to
ln -s ../from/src to/dst

Here is an example, when using python's `os.chdir` function:

[fabien@asus ~]$ python
>>> import os
>>> os.chdir('/tmp/to/dst')
>>> os.system('/usr/bin/pwd')
/tmp/from/src
0
>>> os.system('/usr/bin/pwd -P')
/tmp/from/src
0
>>> os.system('/usr/bin/pwd -L')
/tmp/from/src
0
>>> os.getcwd()
'/tmp/from/src'
>>> 

And here is an example when the folder is first changed with bash:


[fabien@asus ~]$ cd /tmp/to/dst/
[fabien@asus dst]$ python
>>> import os
>>> os.system('/usr/bin/pwd')
/tmp/from/src
0
>>> os.system('/usr/bin/pwd -P')
/tmp/from/src
0
>>> os.system('/usr/bin/pwd -L')
/tmp/to/dst
0
>>> os.getcwd()
'/tmp/from/src'
>>>

--

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29635>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29635] os.chdir() acts like `cd -P` by default, it should offer an option to not follow symlinks

[Fabien Dubosson]

Changes by Fabien Dubosson <fabien.dubos...@gmail.com>:


--
components: Interpreter Core
nosy: StreakyCobra
priority: normal
severity: normal
status: open
title: os.chdir() acts like `cd -P` by default, it should offer an option to 
not follow symlinks
type: behavior
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29635>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Cannot import GDAL

[Fabien]

On 01/11/2017 10:59 PM, Fabien wrote:

I would strongly recommend to use conda
(http://conda.pydata.org/miniconda.html) for managing your packages and
environments, and then use the conda-forge channel
(https://conda-forge.github.io/) to install gdal:

conda install -c conda-forge gdal



I forgot to mention that since py3.6 is quite recent, most packages 
still don't have binaries ready for it. Staying with py3.5 is a better 
idea for now.



--
https://mail.python.org/mailman/listinfo/python-list

Re: Cannot import GDAL

[Fabien]

On 01/11/2017 06:31 PM, Falter, Donald [USA] wrote:

I am new to Python and I am trying to utilize GDAL


GDAL is notoriously difficult to install :-(

I would strongly recommend to use conda 
(http://conda.pydata.org/miniconda.html) for managing your packages and 
environments, and then use the conda-forge channel 
(https://conda-forge.github.io/) to install gdal:


conda install -c conda-forge gdal

Good luck!

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: topology rules in python

[Fabien]

On 12/21/2016 07:11 AM, Bernd Nawothnig wrote:

On 2016-12-20, Xristos Xristoou wrote:

I have a PostGIS database with shapefiles lines, polygons and points
and I want to create a topology rules with python. Any idea how to do
that ?some packages ?

http://www.gdal.org/

or:

pip install gdal


also: shapely, geopandas


--
https://mail.python.org/mailman/listinfo/python-list

Re: The Case Against Python 3

[Fabien]

On 11/25/2016 09:29 AM, Mark Summerfield wrote:

On Thursday, November 24, 2016 at 7:35:03 PM UTC, bream...@gmail.com wrote:

> It's all here https://learnpythonthehardway.org/book/nopython3.html

I think the article is full of factual errors and is at best misleading

> and at worst downright dishonest in its criticisms of Python 3.

> (snipped answers)

Thanks for your answers, which concur well with the post that Mark 
Lawrence linked earlier 
(https://eev.ee/blog/2016/11/23/a-rebuttal-for-python-3/ ).


I'd be interested to read what the community thinks about the fact that 
his book (learn the hard way) is extremely influential among beginners, 
and what tools do we have to avoid that beginners stumble across such 
opinions in their very first steps with python...


--
https://mail.python.org/mailman/listinfo/python-list

Re: Interfacing a dynamic shared library gives me different results in 2.7 versus 3.5

[Fabien]

On 05/23/2016 05:15 AM, Siyi Deng wrote:

I have a dynamic library doing some numerical computations.
I used ctypes to interact it by passing numpy arrays back and forth.
Python 3.5 gives me the correct results.
Python 2.7 gives me different, erroneous results, but it never crashes.
How is this possible? There is no string operations involved whatsoever.


You might be more successful if you describe your problem in more detail 
(maybe with a minimal working example?) and if you ask the scipy/numpy 
people: https://www.scipy.org/scipylib/mailing-lists.html


Cheers
--
https://mail.python.org/mailman/listinfo/python-list

Re: Analytical Geometry in Python with GeoMath

[Fabien]

On 05/04/2016 01:48 AM, Vinicius Mesel wrote:

f you want to check it out, here is the link:https://github.com/vmesel/GeoMath


Thanks! What differences will your package have with shapely?

https://github.com/Toblerity/Shapely

Cheers

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: A pickle problem!

[Fabien]

On 04/21/2016 11:43 PM, Paulo da Silva wrote:

class C(pd.DataFrame):


Note also that subclassing pandas is not always encouraged:

http://pandas.pydata.org/pandas-docs/stable/internals.html#subclassing-pandas-data-structures

Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Strange range

[Fabien]

On 04/01/2016 03:26 PM, Steven D'Aprano wrote:

Incorrect. range is a lazy sequence.


But how does range "know" that it has to start from scratch again? As in 
this example:


it = range(10)
for i in it:
if i >= 3:
break
for i in it:
# why does it start from zero again?
print(i)
--
https://mail.python.org/mailman/listinfo/python-list

Re: Improving performance in matrix operations

[Fabien]

On 03/09/2016 09:09 PM, Drimades wrote:

Another thing to consider is that matrices I'm processing are heavily sparse.


Did you look at scipy.sparse.linalg ?

http://docs.scipy.org/doc/scipy/reference/sparse.linalg.html
--
https://mail.python.org/mailman/listinfo/python-list

Re: Pyhon 2.x or 3.x, which is faster?

[Fabien]

On 03/07/2016 12:38 PM, BartC wrote:


(Although competing with CPython is too easy. PyPy is more of a problem.
With the Jpeg benchmark I mentioned, I can beat PyPy up to 6Mpix, but
then PyPy starts to get faster. At 80Mpix, PyPy is 60% faster.)


Just out of curiosity: are you also competing against numpy/scipy?
--
https://mail.python.org/mailman/listinfo/python-list

Re: urlopen, six, and py2

[Fabien]

On 03/02/2016 03:35 PM, Matt Wheeler wrote:

I agree that six should probably handle this,


Thank you Matt and Chris for your answers. Do you think I should open an 
issue on six? It sounds unlikely that I am the first one having this 
problem...


(until this difference with urlopen I have found six to be extremely 
good at helping not caring about python versions at all)

--
https://mail.python.org/mailman/listinfo/python-list

urlopen, six, and py2

[Fabien]

Hi,

it seems that urlopen had no context manager for versions < 3. The 
following code therefore will crash on py2 but not on py3.


from six.moves.urllib.request import urlopen
with urlopen('http://www.google.com') as resp:
_ = resp.read()

Error:
AttributeError: addinfourl instance has no attribute '__exit__'

I actually wonder if this is not something that the six library should 
take care of upstream, but in the meantime I could simply do what is 
suggested on this stackoverflow post:


http://stackoverflow.com/questions/30627937/tracebaclk-attributeerroraddinfourl-instance-has-no-attribute-exit

My question is: why does the python3 version need a "with" block while 
the python2 version doesn't? Can I skip the "with" entirely, or should I 
rather do the following:


from six.moves.urllib.request import urlopen

try:
with urlopen('http://www.google.com') as resp:
_ = resp.read()
except AttributeError:
# python 2
resp = urlopen('http://www.google.com')
_ = resp.read()

(which is quite ugly).

Thanks!

Fabien



--
https://mail.python.org/mailman/listinfo/python-list

Re: Problem

[Fabien]

On 12/08/2015 04:03 PM, Namrah Anwar wrote:

Dear Administration,

I am Namrah Anwar writing to you from Pakistan. I downloaded Python version
3.5.1 and 2.7 and after installation at first, upon opening again it asked

(snip)

-- *Regards,Namrah Anwar* *PhD Student (Fellowship) - Cancer Biology -


I can see from your signature that you are going to use python for 
scientific purposes. I recommend to go directly for Anaconda:


https://www.continuum.io/downloads

Cheers,

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: Plotting timeseries from a csv file using matplotlib

[Fabien]

On 11/13/2015 06:27 AM, Karthik Sharma wrote:

How do I extract all the values in columns


Have a look at pandas: 
http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html

--
https://mail.python.org/mailman/listinfo/python-list

Re: Is it ok to install python binaries on a network drive?

[Fabien]

On 10/29/2015 01:07 PM, Steven D'Aprano wrote:

My question is: what will happen when about 20 students will do the same
>at the same time? I know too little about what's going under the hood
>with an "import X" command to be able to predict problems that might
>occur.

I think it should be fine.


Dear all,

I just wanted to report that everything worked fine. It wasn't even too 
slow. I find it a quite elegant solution after all for all the students 
to have the same working environment.


Thanks!

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3 sort() problem

[Fabien]

On 08/17/2015 01:42 PM, Владислав wrote:

x = [1, 2, 4, 2, 1, 3]
x = list(set(x)).sort()
print(x) /# output: None/

I know that sort() returns None, but I guess that it would be returned x
that was sorted. Why so?


If sort() returns None, than the following:

x = list(set(x)).sort()

is equivalent to writing:

x = None

So I don't really understand your question, I'm sorry...

Cheers,

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: How to construct matrix from vectors?

[Fabien]

On 06/21/2015 07:21 AM, Nasser M. Abbasi wrote:

v1=np.array([(1,2,3)]).T
v2=np.array([(4,5,6)]).T
v3=np.array([(7,8,9)]).T
v4=np.array([(10,11,12)]).T

mat =np.hstack(( np.vstack((v1,v3)), np.vstack((v2,v4))) )

Out[236]:
array([[ 1,  4],
[ 2,  5],
[ 3,  6],
[ 7, 10],
[ 8, 11],
[ 9, 12]])

There are way too many '(([[]]))' things  in Python :)


another solution with less (([[]])), and less ;. There are way too 
many ; in Matlab ;)


import numpy as np
v1 = [1, 2, 3]
v2 = [4, 5, 6]
v3 = [7, 8, 9]
v4 = [10, 11, 12]
np.hstack([[v1, v2], [v3, v4]]).T
Out[]:
array([[ 1,  4],
   [ 2,  5],
   [ 3,  6],
   [ 7, 10],
   [ 8, 11],
   [ 9, 12]])

--
https://mail.python.org/mailman/listinfo/python-list

Re: Catching exceptions with multi-processing

[Fabien]

On 06/19/2015 10:58 PM, Chris Angelico wrote:

AIUI what he's doing is all the subparts of task1 in parallel, then
all the subparts of task2:

pool.map(task1, dirs, chunksize=1)
pool.map(task2, dirs, chunksize=1)
pool.map(task3, dirs, chunksize=1)

task1 can be done on all of dirs in parallel, as no instance of task1
depends on any other instance of task1; but task2 should be started
only if all task1s finish successfully. OP, is this how it is? If not,
I apologize for the noise.


That's it! Thanks for clarifying, I might have trouble explaining myself 
sometimes ;-)


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Catching exceptions with multi-processing

[Fabien]

On 06/20/2015 05:14 AM, Cameron Simpson wrote:

I would keep your core logic Pythonic, raise exceptions. But I would
wrap each task in something to catch any Exception subclass and report
back to the queue. Untested example:

  def subwrapper(q, callable, *args, **kwargs):
try:
  q.put( ('COMPLETED', callable(*args, **kwargs)) )
except Exception as e:
  q.put( ('FAILED', e, callable, args, kwargs) )

then dispatch tasks like this:

  pool.map(subwrapper, q, task1, dirs, chunksize=1)

and have a thread (or main program) collect things from the queue for
logging and other handling. Obviously you might return something more
sophisticated that my simple tuple above, but I'm sure you get the idea.

Cheers,
Cameron Simpson


Perfect! Much more elegant and easier to implement on top of my existing 
workflow based on raising exceptions.


thanks to all responses,

Fabien



--
https://mail.python.org/mailman/listinfo/python-list

Catching exceptions with multi-processing

[Fabien]

Folks,

I am developing a tool which works on individual entities (glaciers) and 
do a lot of operations on them. There are many tasks to do, one after 
each other, and each task follows the same interface:


def task_1(path_to_glacier_dir):
open file1 in path_to_glacier_dir
do stuff
if dont_work:
raise RuntimeError(didnt work)
write file2 in path_to_glacier_dir

This way, the tasks can be run in parallel very easily:

import multiprocessing as mp
pool = mp.Pool(4)

dirs = [list_of_dirs]
pool.map(task1, dirs, chunksize=1)
pool.map(task2, dirs, chunksize=1)
pool.map(task3, dirs, chunksize=1)

... and so forth. I tested the tool for about a hundred glaciers but now 
it has to run for thousands of them. There are going to be errors, some 
of them are even expected for special outliers. What I would like the 
tool to do is that in case of error, it writes the identifier of the 
problematic glacier somewhere, the error encountered and more info if 
possible. Because of multiprocessing, I can't write in a shared file, so 
I thought that the individual processes should write a unique error 
file in a dedicated directory.


What I don't know how to, however, is how to do this at minimal cost and 
in a generic way for all tasks. Also, the task2 should not be run if 
task1 threw an error. Sometimes (for debugging), I'd rather keep the 
normal behavior of raising an error and stopping the program.


Do I have to wrap all tasks with a try: exept: block? How to switch 
between behaviors? All the solutions I could think about look quite ugly 
to me. And it seems that this is a general problem that someone cleverer 
than me had solved before ;-)


Thanks,

Fabien







--
https://mail.python.org/mailman/listinfo/python-list

Re: Catching exceptions with multi-processing

[Fabien]

On 06/19/2015 05:41 PM, Steven D'Aprano wrote:

On Sat, 20 Jun 2015 12:01 am, Fabien wrote:


Folks,

I am developing a tool which works on individual entities (glaciers) and
do a lot of operations on them. There are many tasks to do, one after
each other, and each task follows the same interface:

I'm afraid your description is contradictory. Here you say the tasks run one
after another, but then you say:


This way, the tasks can be run in parallel very easily:

and then later still you contradict this:


Also, the task2 should not be run if task1 threw an error.


If task2 relies on task1, then you*cannot*  run them in parallel. You have
to run them one after the other, sequentially.


Hi Steve,

I meant: for several glaciers in parallel as shown by the code snippet:

import multiprocessing as mp
pool = mp.Pool(4)

dirs = [list_of_dirs]
pool.map(task1, dirs, chunksize=1)
pool.map(task2, dirs, chunksize=1)
pool.map(task3, dirs, chunksize=1)

which should be changed to something like (after some of the responses):

dirs = [list_of_dirs]
pool.map(task1, dirs, ...)
# handle exceptions
dirs_reduced = [dirs that did not fail]
pool.map(task2, dirs_reduced, ...)

this way the tasks are run sequentially for each glacier but in parallel 
over a list of glaciers...


Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: Catching exceptions with multi-processing

[Fabien]

On 06/19/2015 04:25 PM, Andres Riancho wrote:

Fabien,

My recommendation is that you should pass some extra arguments to the task:
 * A unique task id
 * A result multiprocessing.Queue

 When an exception is raised you put (unique_id, exception) to the
queue. When it succeeds you put (unique_id, None). In the main process
you consume the queue and do your error handling.

 Note that some exceptions can't be serialized, there is where
tblib [0] comes handy.

[0]https://pypi.python.org/pypi/tblib

Regards,


Thanks, I wasn't aware of the multiprocessing.Queue workflow. It seems 
like its going to require some changes in the actual code of the tasks 
though. Did I get it right that I should stop raising exceptions then?


Something like:

def task_1(path, q):
# Do stuffs
if dont_work:
q.put(RuntimeError(didnt work))
return
# finished
q.put(None)
return


Fabien





--
https://mail.python.org/mailman/listinfo/python-list

Re: Classic OOP in Python

[Fabien]

On 06/17/2015 11:16 PM, sohcahto...@gmail.com wrote:

You don't need interfaces with Python.  Duck typing makes that all possible.


Yes, but I also like interfaces (or in python: mimicked interfaces with 
NotImplementedError) for their clarity and documentation purposes.


Would you consider the following kind of program unpythonic?

class MovingObject(object):
Great doc about what a moving object is

def move(self):
Great doc about move
raise NotImplementedError()

class Dog(MovingObject):
def move(self):
print Dog is moving

class Car(MovingObject):
def move(self):
print Car is moving

(Disclaimer: I learned OOP with Java)

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: zip as iterator and bad/good practices

[Fabien]

On 06/12/2015 05:26 PM, Ian Kelly wrote:

but that awful bug made me wonder: is it a bad practice to
interactively modify the list you are iterating over?

Generally speaking, yes, it's bad practice to add or remove items
because this may result in items being visited more than once or not
at all. Modifying or replacing items however is usually not an issue.



Thanks. In that case I was modifying items and needed them to be updated 
during the loop. I kept the solution as is and my tests pass in 2 and 3.


I will consider using six. Currently all my modules begin with:


from __future__ import division
try:
from itertools import izip as zip
except ImportError:
pass

Which might even become longer if I find other bugs ;-)

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

zip as iterator and bad/good practices

[Fabien]

Folks,

I am developing a program which I'd like to be python 2 and 3 
compatible. I am still relatively new to python and I use primarily py3 
for development. Every once in a while I use a py2 interpreter to see if 
my tests pass through.


I just spent several hours tracking down a bug which was related to the 
fact that zip is an iterator in py3 but not in py2. Of course I did not 
know about that difference. I've found the izip() function which should 
do what I want, but that awful bug made me wonder: is it a bad practice 
to interactively modify the list you are iterating over?


I am computing mass fluxes along glacier branches ordered by 
hydrological order, i.e. branch i is guaranteed to flow in a branch 
later in that list. Branches are objects which have a pointer to the 
object they are flowing into.


In pseudo code:

for stuff, branch in zip(stuffs, branches):
# compute flux
...
# add to the downstream branch
id_branch = branches.index(branch.flows_to)
branches[id_branch].property.append(stuff_i_computed)

So, all downstream branches in python2 where missing information from 
their tributaries. It is quite a dangerous code but I can't find a more 
elegant solution.


Thanks!

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: zip as iterator and bad/good practices

[Fabien]

On 06/12/2015 05:00 PM, Fabien wrote:

I've found the izip() function which should do what I want


I've just come accross a stackoverflow post where they recommend:

from future_builtins import zip

which is OK since I don't want to support versions = 2.6
--
https://mail.python.org/mailman/listinfo/python-list

Re: zip as iterator and bad/good practices

[Fabien]

On 06/12/2015 05:26 PM, Ian Kelly wrote:

for stuff, branch in zip(stuffs, branches):
 # compute flux
 ...
 # add to the downstream branch
 id_branch = branches.index(branch.flows_to)
 branches[id_branch].property.append(stuff_i_computed)

Er, I don't see the problem here. The branch object in the zip list
and the branch object in branches should be the*same*  object, so the
downstream branch update should be reflected when you visit it later
in the iteration, regardless of whether zip returns a list or an iterator.

Tangentially, unless you're using id_branch for something else that
isn't shown here, is it really necessary to search the list for the
downstream branch when it looks like you already have a reference to
it? Could the above simply be replaced with:

 branch.flows_to.property.append(stuff_i_computed)


Thanks a lot for your careful reading! I overly simplified my example 
and indeed this line works fine. I was adding things to stuffs too, 
which is a list of lists... Sorry for the confusion!

--
https://mail.python.org/mailman/listinfo/python-list

Re: So what's happening here?

[Fabien]

On 06/05/2015 02:46 PM, Paul Appleby wrote:

I saw somewhere on the net that you can copy a list with slicing. So
what's happening when I try it with a numpy array?


Python lists and numpy arrays are NOT the same thing. This is one of the 
reasons why numpy was developed in the first place. Numpy uses views, 
not copies. There is however a .copy() method on numpy arrays


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Knowledge Requirements for Machine Learning (SKLearn)

[Fabien]

On 06/02/2015 11:16 AM, chrismeek4...@gmail.com wrote:

I would like to get into Machine learning.


An Introduction to Statistical Learning:
http://www-bcf.usc.edu/~gareth/ISL/

and corresponding MOOC.

SKLearn and Statsmodels mailing lists:
http://dir.gmane.org/gmane.comp.python.scikit-learn
http://dir.gmane.org/gmane.comp.python.pystatsmodels

Cheers,

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: Slices time complexity

[Fabien]

On 05/18/2015 09:49 PM, Ian Kelly wrote:

It may be possible that lists in CPython could be made to share their
internal arrays with other lists on a copy-on-write basis, which could
allow slicing to be O(1) as long as neither list is modified while the
array is being shared. I expect this would be a substantial piece of
work, and I don't know if it's something that anybody has looked into.


Isn't Numpy doing this (not sure, not a python nor a numpy expert):

 import numpy as np
 a = np.array([1,2,3,4])
 b = a[1:]
 b[0] = 9
 a
array([1, 9, 3, 4])


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Pickle based workflow - looking for advice

[Fabien]

On 14.04.2015 06:05, Chris Angelico wrote:

Not sure what you mean, here. Any given file will be written by
exactly one process? No possible problem. Multiprocessing within one
application doesn't change that.


yes that's what I meant. Thanks!
--
https://mail.python.org/mailman/listinfo/python-list

Re: Pickle based workflow - looking for advice

[Fabien]

On 13.04.2015 17:45, Devin Jeanpierre wrote:

On Mon, Apr 13, 2015 at 10:58 AM, Fabienfabien.mauss...@gmail.com  wrote:

Now, to my questions:
1. Does that seem reasonable?

A big issue is the use of pickle, which is:

* Often suboptimal performance wise (e.g. you can't load only subsets
of the data)
* Makes forwards/backwards compatibility very difficult
* Can make python 2/3 migrations harder
* Creates data files which are difficult to analyze/fix by hand if
they get broken
* Is schemaless, and can accidentally include irrelevant data you
didn't mean to store, making all of the above worse.
* Means you have to be very careful who wrote the pickles, or you open
a remote code execution vulnerability. It's common for people to
forget that code is unsafe, and get themselves pwned. Security is
always better if you don't do anything bad in the first place, than if
you do something bad but try to manage the context in which the bad
thing is done.

Cap'n Proto might be a decent alternatives that gives you good
performance, by letting you process only the bits of the file you want
to. It is also not a walking security nightmare.


Thanks for your thoughts. All these concerns are rather secondary for 
the kind of tool I am working on, with the exception of speed. I will 
have a look at Proto






2. Should Watershed be an object or should it be a simple dictionary? I
thought that an object could be nice, because it could take care of some
operations such as plotting and logging. Currently I defined a class
Watershed, but its attributes are defined and filled by A, B and C (this
seems a bit wrong to me).

It is usually very confusing for attributes to be defined anywhere
other than __init__. It's very really confusing for them to be defined
by some random other function living somewhere else.


Yes, OK. I will stop that.


I could give more responsibilities to this class
but it might become way too big: since the whole purpose of the tool is to
work on watersheds, making a Watershed class actually sounds like a code
smell (http://en.wikipedia.org/wiki/God_object)

Whether they are methods or not doesn't make this any more or less of
a god object -- if it stores all this data used by all these different
things, it is already a bit off.


Yes, but I see no other way. The god container will probably be the 
watershed's directory with the data in it. The rest will specialize.




3. The operation A opens an external file, reads data out of it and writes
it in Watershed object. Is it a bad idea to multiprocess this? (I guess it
is, since the file might be read twice at the same time)

That does sound like a bad idea, for the reason you gave. It might be
possible to read it once, and share it among many processes.


Yes. Thanks!

--
https://mail.python.org/mailman/listinfo/python-list

Re: Pickle based workflow - looking for advice

[Fabien]

On 13.04.2015 19:08, Peter Otten wrote:

How about a file-based workflow?

Write distinct scripts, e. g.

a2b.py that reads from *.a and writes to *.b

and so on. Then use a plain old makefile to define the dependencies.
Whether .a uses pickle, .b uses json, and .z uses csv is but an
implementation detail that only its producers and consumers need to know.
Testing an arbitrary step is as easy as invoking the respective script with
some prefabricated input and checking the resulting output file(s).


I think I like the idea because it is more durable. The data I 
manipulate comes with specific formats which are very efficient. With 
the pickle I was kind of lazy and, well, saved a couple of read/write 
routines.


Still, your idea is probably more elegant.

With multiprocessing, do I have to care about processes writing 
simultaneously in *different* files? I guess the OS takes good care of 
this stuff but I'm not an expert.


Tahnks,

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: Pickle based workflow - looking for advice

[Fabien]

On 13.04.2015 18:25, Dave Angel wrote:

On 04/13/2015 10:58 AM, Fabien wrote:

Folks,



A comment.  Pickle is a method of creating persistent data, most
commonly used to preserve data between runs.  A database is another
method.  Although either one can also be used with multiprocessing, you
seem to be worrying more about the mechanism, and not enough about the
problem.


I am writing a quite extensive piece of scientific software. Its
workflow is quite easy to explain. The tool realizes series of
operations on watersheds (such as mapping data on it, geostatistics and
more). There are thousands of independent watersheds of different size,
and the size determines the computing time spent on each of them.


First question:  what is the name or identity of a watershed?
Apparently it's named by a directory.  But you mention ID as well.  You
write a function A() that takes only a directory name. Is that the name
of the watershed?  One per directory?  And you can derive the ID from
the directory name?

Second question, is there any communication between watersheds, or are
they totally independent?

Third:  this external data, is it dynamic, do you have to fetch it in
a particular order, is it separated by watershed id, or what?

Fourth:  when the program starts, are the directories all empty, so the
presence of a pickle file tells you that A() has run?  Or is there some
other meaning for those files?



Say I have the operations A, B, C and D. B and C are completely
independent but they need A to be run first, D needs B and C, and so
forth. Eventually the whole operations A, B, C and D will run once for
all,


For all what?


but of course the whole development is an iterative process and I
rerun all operations many times.


Based on what?  Is the external data changing, and you have to rerun
functions to update what you've already stored about them?  Or do you
just mean you call the A() function on every possible watershed?



(I suddenly have to go out, so I can't comment on the rest, except that
choosing to pickle, or to marshall, or to database, or to
custom-serialize seems a bit premature.  You may have it all clear in
your head, but I can't see what the interplay between all these calls to
one-letter-named functions is intended to be.)



Thanks Dave for your interest. I'll make an example:

external files:
- watershed outlines (single file)
- global topography (single file)
- climate data (single file)

Each watershed has an ID. Each watershed is completely independant.

So the function A for example will take one ID as argument, open the 
watershed file and extract its outlines, make a local map, open the 
topography file, extract a part of it, make a watershed object and store 
the watersheds local data in it.


Function B will open the watershed pickle, take the local information it 
needs (like local topography, already cropped to the region of interest) 
and map climate data on it.


And so forth, so that each function A, B, C, ... builds upon the 
information of the others and adds it's own service in terms of data.


Currently, all data (numpy arrays and vecor objects mostly) are stored 
as object attributes, which is I guess bad practice. It's kind of a 
database for dummies: read topography of watershed ID 0128 will be:

- open watershed.p in the '0128' directory
- read the watershed.topography attribute

I think that I like Peter's idea to follow a file based workflow 
instead, and forget about my watershed object for now.


But I'd still be interested in your comments if you find time for it.

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Pickle based workflow - looking for advice

[Fabien]

Folks,

I am writing a quite extensive piece of scientific software. Its 
workflow is quite easy to explain. The tool realizes series of 
operations on watersheds (such as mapping data on it, geostatistics and 
more). There are thousands of independent watersheds of different size, 
and the size determines the computing time spent on each of them.


Say I have the operations A, B, C and D. B and C are completely 
independent but they need A to be run first, D needs B and C, and so 
forth. Eventually the whole operations A, B, C and D will run once for 
all, but of course the whole development is an iterative process and I 
rerun all operations many times.


Currently my workflow is defined as follows:

Define a unique ID and file directory for each watershed, and define A 
and B:


def A(watershed_dir):
# read some external data
# do stuff
# Store the stuff in a Watershed object
# save it
f_pickle = os.path.join(watershed_dir, 'watershed.p')
with open(f_pickle, 'wb') as f:
pickle.dump(watershed, f)

def B(watershed_directory):
w = pickle.read()
f_pickle = os.path.join(watershed_dir, 'watershed.p')
with open(f_pickle, 'rb') as f:
watershed = pickle.load(f)
# do new stuff
# store it in watershed and save
with open(f_pickle, 'wb') as f:
pickle.dump(watershed, f)

So the watershed object is a data container which grows in content. The 
pickle that stores the info can reach a few Mb of size. I chose this 
strategy because A, B, C and D are independent, but they can share their 
results through the pickle. The functions have a single argument (the 
path to the working directory), which means that when I run the 
thousands catchments I can use the multiprocessing pool:


import multiprocessing as mp
poolargs = [list of directories]
pool = mp.Pool()
poolout = pool.map(A, poolargs, chunksize=1)
poolout = pool.map(B, poolargs, chunksize=1)
etc.

I can easily choose to rerun just B without rerunning A. Reading and 
writing pickle times is real slow in comparison to the other stuffs to 
do (running B or C on a single catchment can take seconds for example).


Now, to my questions:
1. Does that seem reasonable?
2. Should Watershed be an object or should it be a simple dictionary? I 
thought that an object could be nice, because it could take care of some 
operations such as plotting and logging. Currently I defined a class 
Watershed, but its attributes are defined and filled by A, B and C (this 
seems a bit wrong to me). I could give more responsibilities to this 
class but it might become way too big: since the whole purpose of the 
tool is to work on watersheds, making a Watershed class actually sounds 
like a code smell (http://en.wikipedia.org/wiki/God_object)
3. The operation A opens an external file, reads data out of it and 
writes it in Watershed object. Is it a bad idea to multiprocess this? (I 
guess it is, since the file might be read twice at the same time)

4. Other comments you might have?

Sorry for the lengthy mail but thanks for any tip.

Fabien





--
https://mail.python.org/mailman/listinfo/python-list

Re: Georeferencing Image with GDAL

[Fabien]

On 19.03.2015 09:00, Sumesh K.C. wrote:

I've been trying to georeference image with GDAL.
(...)
I'm trying to do everything for the beginning.

Can anyone guide me, how i should start?


Hi,

first I would try to do some tutorials about GDAL and read some 
documentation and examples. Then I would try something and if it doesn't 
work and I really don't understand why I would ask for help in the 
python GIS community: http://blog.gmane.org/gmane.comp.python.gis By 
doing so I would try to be as precise as possible in formulating my 
question.


All the best,

Fabien


--
https://mail.python.org/mailman/listinfo/python-list

Re: date

[Fabien]

On 02.03.2015 12:55, greymausg wrote:

I have a csv file, the first item on a line is the date in the format
2015-03-02 I try to get that as a date by date(row[0]), but it barfs,
replying Expecting an integer. (I am really trying to get the offset
in weeks from that date to today())


Have you tried Pandas? http://pandas.pydata.org/

If your csv file has no other problems, the following should do the trick:

import pandas as pd
df = pd.read_csv('file.csv', index_col=0, parse_dates= {time : [0]})

Cheers,

Fabien




--
https://mail.python.org/mailman/listinfo/python-list

Re: suggestions for functional style (singleton pattern?)

[Fabien]

On 01.03.2015 06:05, Michael Torrie wrote:

  A module*is*  a singleton pattern, particularly one
that maintains state.  I use sometimes use this feature for sharing
config and other data between other modules (global state when it's
required).


I do this too, after some helping recommendations I got from this 
discussion group. I find it neat to use a module for sharing config 
states, but I always wondered: does this pattern fall into the globals 
are devil category?


Cheers,

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: date

[Fabien]

On 02.03.2015 15:26, Mark Lawrence wrote:


Have you tried Pandas? http://pandas.pydata.org/

If your csv file has no other problems, the following should do the
trick:

import pandas as pd
df = pd.read_csv('file.csv', index_col=0, parse_dates= {time : [0]})

Cheers,

Fabien



IMHO complete overkill.  Give me the Steven D'Aprano solution any day of
the week.


Without knowing anything about the OP background, I still hope my 
suggestion is a good one. Pandas is one of the best thing that happened 
to me in my python life, I'm happy to at least suggest it. But yeah, if 
you just want to read the csv and do no data crunching on it, pandas is 
overkill.


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: iPython 3 released

[Fabien]

On 01.03.2015 19:24, Mark Lawrence wrote:

This seemed to be a low key affair, at least from my perspective


True, no messages yet on the scientific python mailing lists either. I 
am wondering if avoiding a massive switch to a *.0 release was the 
reason for this (relative) silence.


Fabien


--
https://mail.python.org/mailman/listinfo/python-list

Re: Gaussian process regression

[Fabien]

On 27.02.2015 18:55, Peter Pearson wrote:

On Thu, 26 Feb 2015 09:59:45 -0800 (PST),jaykim.hui...@gmail.com  wrote:


I am trying to use Gaussian process regression for Near Infrared
spectra. I have reference data(spectra), concentrations of reference
data and sample data, and I am trying to predict concentrations of
sample data. Here is my code.

from sklearn.gaussian_process import GaussianProcess
gp = GaussianProcess()
gp.fit(reference, concentration)
concentration_pred = gp.predict(sample)

[snip]

I'm sorry you're not getting help from this normally very helpful group.
I'd guess that's because nobody here uses sklearn.  Where did you get
sklearn?  Is it possible that there's an sklearn forum somewhere?


http://blog.gmane.org/gmane.comp.python.scikit-learn

Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: pypandoc and restructured text

[Fabien]

On 22.02.2015 00:20, alb wrote:

I finally upgraded! And I'm currently trying out xfce!
Thanks again for the suggestions.

Al

p.s.: now pandoc works as expected.


I don't want to sound insistent, but as a Linux user I personnaly 
recommend not to use apt to install and use python packages. Installed 
packages will soon or late become outdated or they will become a burden 
because you have less control on what you have installed and why.


I really like virtualenv for it's help in keeping things (system / 
python version / fooling around with new packages) separated. It's also 
the recommendation of the recommendation of the Python Packaging 
Authority:

https://packaging.python.org/en/latest/current.html#installation-tool-recommendations

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: pypandoc and restructured text

[Fabien]

On 21.02.2015 14:39, alb wrote:

Do you know of anyway to install wheezy packages on squeeze?


No need for a new distro. Use virtualenv, this is simple and great:

http://simononsoftware.com/virtualenv-tutorial-part-2/

Cheers,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python discussed in Nature

[Fabien]

On 12.02.2015 10:31, wxjmfa...@gmail.com wrote:
[some OT stuffs about unicode]

... what a coincidence then that a huge majority of scientists 
(including me) dont care AT ALL about unicode. But since scientists are 
not paid to rewrite old code, the scientific world is still stuck to 
python 2. It's a pitty, given how easy it is to write py2/py3 compatible 
scientific tools.


Thanks for the link to the article steven!

Fabien

(sorry for the OT  sorry for feeding the t)
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python discussed in Nature

[Fabien]

On 12.02.2015 12:25, Marko Rauhamaa wrote:

Fabienfabien.mauss...@gmail.com:


... what a coincidence then that a huge majority of scientists
(including me) dont care AT ALL about unicode.

You shouldn't, any more than you care about ASCII or 2's-complement
encoding. Things should just work.


And they do! Since almost one year in writing scientific python code, 
not a single problem. I wouldnt even know about issues if i didnt read 
some posts here.



But since scientists are not paid to rewrite old code, the scientific
world is still stuck to python 2. It's a pitty, given how easy it is
to write py2/py3 compatible scientific tools.

What's a pity is that Python3 chose to ignore the seamless transition
path. It would have been nice, for example, to have all Python 3 code
explicitly mark its dialect (a .py3 extension, a magic import or
something) and then allow legacy Py2 code and Py3 code coexist the same
way C and C++ can coexist.


But this was exactly my point! Today in 2015 it's incredibly easy to 
write py2/py3 code for a scientist. The whole SciPy track has done the 
transition. Not an issue anymore either, for me at least (python 
youngster ;-)


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: idle feature request

[Fabien]

On 11.02.2015 14:11, Rustom Mody wrote:

Context:
I am using idle for taking python classes.


I know this is not your question, but: have you considered using iPython 
notebooks for teaching? They save automatically, look nice, and can be 
sent as code, html, rst, ...


Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: idle feature request

[Fabien]

On 11.02.2015 14:29, Rustom Mody wrote:

So getting up and running with minimal headache is an important priority.


Yes true. I consider Notebooks as a way to do frontal teaching, not 
necessarily usefull if students have to redo what you are doing on 
screen. For that, notebooks are rather prone to confusion because the 
order with wich you run the cells is important, so total beginners are 
really confused about variables overwritting etc.


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

'pip install nose' fails on python3 virtualenv

[Fabien]

Folks,

I couldn't find any info on this problem online:

Linux Mint
Python 3.3
virtualenv (12.0.7)
pip (6.0.8)
setuptools (12.0.5)

(testpy3)mowglie@flappi ~ $ pip install nose
Collecting nose
  Exception:
  Traceback (most recent call last):
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/basecommand.py, 
line 232, in main

  status = self.run(options, args)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/commands/install.py, 
line 339, in run

  requirement_set.prepare_files(finder)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/req/req_set.py, 
line 333, in prepare_files

  upgrade=self.upgrade,
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/index.py, 
line 305, in find_requirement

  page = self._get_page(main_index_url, req)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/index.py, 
line 783, in _get_page

  return HTMLPage.get_page(link, req, session=self.session)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/index.py, 
line 872, in get_page

  Cache-Control: max-age=600,
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/requests/sessions.py, 
line 473, in get

  return self.request('GET', url, **kwargs)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/download.py, 
line 365, in request

  return super(PipSession, self).request(method, url, *args, **kwargs)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/requests/sessions.py, 
line 461, in request

  resp = self.send(prep, **send_kwargs)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/requests/sessions.py, 
line 573, in send

  r = adapter.send(request, **kwargs)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/cachecontrol/adapter.py, 
line 36, in send

  cached_response = self.controller.cached_request(request)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/cachecontrol/controller.py, 
line 100, in cached_request

  resp = self.serializer.loads(request, self.cache.get(cache_url))
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/cachecontrol/serialize.py, 
line 108, in loads

  return getattr(self, _loads_v{0}.format(ver))(request, data)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/cachecontrol/serialize.py, 
line 148, in _loads_v1

  return self.prepare_response(request, cached)
File 
/home/mowglie/.pyvirtualenvs/testpy3/lib/python3.3/site-packages/pip/_vendor/cachecontrol/serialize.py, 
line 129, in prepare_response

  body = io.BytesIO(cached[response].pop(body))
  TypeError: 'str' does not support the buffer interface

I'm quite sure this problem appeared recently, I did not make any change 
to my system recently...


Thanks,

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

What does pip install do?

[Fabien]

Folks,

I've learned a lot today about python packaging and stuff, thanks to a 
couple of good websites. I've learned how to install a package from PyPi 
with pip, choose which version, upgrade it, uninstall it, use 
virtualenv, BUT I'm still asking myself: what does pip install 
*concretely* do on my virtual environment?


It seems to add a directory in the /site-packages directory, but what 
does it do when for example command-line scripts are shipped with it? Is 
something else happening in the background?


Neither does the Python Packaging Authority website 
(https://pip.pypa.io/en/latest/user_guide.html) nor the 
python-packaging-user-guide 
(http://python-packaging-user-guide.readthedocs.org/en/latest/installing.html) 
nor the Installing python packages from the python docs 
(https://docs.python.org/3/installing/) seem to provide an answer to 
this question.


Or did I simply miss it?

Thanks!

Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: What does pip install do?

[Fabien]

On 12.01.2015 23:46, Chris Angelico wrote:

On Tue, Jan 13, 2015 at 9:35 AM, Fabienfabien.mauss...@gmail.com  wrote:

BUT I'm still asking myself: what does pip install*concretely*  do on my
virtual environment?

As far as I know, it's equivalent to three steps:

1) Download the appropriate version of a package (the latest, if you
didn't say otherwise)
2) Extract that package
3) Run 'python setup.py'.

What setup.py does is up to the package, but for a pure-Python package
it should simply create something in site-packages. It might do a lot
more, though - compiling C or Fortran code, for instance.


Thanks Chris. I should then rephrase my question to what does python 
setup.pt do? ;-). My point was also that I think that this information 
(mostly: what will change on my system or my virtual env when I install 
a python package?) could/should be provided in the mentioned resources:


Python Packaging Authority website 
(https://pip.pypa.io/en/latest/user_guide.html),
python-packaging-user-guide 
(http://python-packaging-user-guide.readthedocs.org/en/latest/installing.html) 

Installing python packages from the python docs 
(https://docs.python.org/3/installing/)


I've found a bit more about this (How installation works) in the 
Python 3 doc which is now considered legacy and might therefore not be 
read:

https://docs.python.org/3.4/install/index.html#installing-python-modules-legacy-version

Fabien




--
https://mail.python.org/mailman/listinfo/python-list

Re: Question about uninstallation.

[Fabien]

On 28.09.2014 03:07, Gregory Johannes-Kinsbourg wrote:

  both Python 2  3 (I’m on OS X 10.10 btw) and first of all was curious to 
know if they will clash


I am also quite new to the python business, and had the same kind of 
questions (how to install/uninstall a package, will different versions 
clash, should I use pip install or pip3 install, etc). And then I 
discovered virtualenv and virtualenvwrapper and everything was much 
easier. Here is a resource that helped me:


http://simononsoftware.com/virtualenv-tutorial-part-2/

I don't know about mac but on linux it works like a charm

Fabien

--
https://mail.python.org/mailman/listinfo/python-list

Re: Linux distros w/o Python in base installation

[Fabien]

On 12.08.2014 09:59, David Palao wrote:

Also Gentoo uses Python3 by default for some months now. The positive
side effect for me has been that I started seriously to switch to
python3.


it's a matter of months for debian/ubuntu to rely only on python3 code 
too, while still respecting PEP 394:


https://wiki.ubuntu.com/Python/3


--
https://mail.python.org/mailman/listinfo/python-list

Re: The right way to use config files

[Fabien]

On 10.08.2014 00:30, Terry Reedy wrote:

The advantage of TDD is that it forces one to make code testable as you do.


Thanks a lot, Terry, for your comprehensive example!
--
https://mail.python.org/mailman/listinfo/python-list

The right way to use config files

[Fabien]

Folks,

I am not a computer scientist (just a scientist) and I'd like to ask 
your opinion about a design problem I have. It's not that I can't get my 
program to work, but rather that I have trouble to find an elegant 
solution to the problem of passing information to my program's elements. 
I have trouble to be clear in my request so my apologies for the long 
text...


The tool I am developing is a classical data-analysis workflow. Ideally, 
all the program's configurations are located in a single .cfg file which 
I parse with ConfigObg. The file contains I/O informations 
(path_to_input, path_to_output) as well as internal options 
(use_this_function, dont_use_this_one, function1_paramx = y), etc...


Currently, my program is a super-object which is initialized once and 
work schematically as follows:


main():
obj = init_superobj(config file)
obj.preprocess()
obj.process()
obj.write()

the superobj init routine parses the config files and reads the input data.

and a processing step can be, for example:

def process():

if self.configfile.as_bool('do_funcion1'):
params = config.parse_function1_params()
call_function1(self.data, params)

if self.configfile.as_bool('do_funcion2'):
params = config.parse_function2_params()
call_function2(self.data2, params)

The functions themselves do not know about the superobject or about the 
configfile. They are standalone functions which take data and 
parameters as input and produce output with it. I thought that the 
standalone functions will be clearer and easier to maintain, since they 
do not rely on some external data structure such as the configobj or 
anything.


BUT, my problem is that several options really are universal options 
to the program, such as the output directory for example. This 
information (where to write their results) is given to most of the 
functions as parameter.


So I had the idea to define a super-object which parses the config file 
and input data and is given as a single parameter to the processing 
functions, and the functions take the information they need from it. 
This is tempting because there is no need for refactoring when I decide 
to change something in the config, but I am afraid that the program may 
become unmaintainable by someone else than myself. Another possibility 
would be at least to give all the functions access to the configfile.


To get to the point: is it good practice to give all elements of a 
program access to the configfile and if yes, how is it done properly?


I hope at least someone will understand what I mean ;-)

Cheers and thanks,

Fabien









--
https://mail.python.org/mailman/listinfo/python-list

Re: The right way to use config files

[Fabien]

Hi Ben,

On 09.08.2014 14:17, Ben Finney wrote:

Have one module of your application be responsible for the configuration
of the application::

 # app/config.py

 import configparser

 parser = configparser.ConfigParser()
 parser.read(app.conf)


Thanks for the suggestion. This way to do is new to me, and I didn't 
come to the idea myself. It seems like a good way to do this. But how to 
give an argument to this config namespace? i.e I want app.conf to be 
given as argument.


Currently my program starts like this:

def main():

# See if the user gave a configfile
if len(sys.argv) == 2:
# file was given as argument
cfg = str(sys.argv[1])
else:
# default file taken in the resource directory
cfg = os.path.abspath(os.path.join(os.path.dirname(__file__),
  os.pardir,'res','default.cfg'))

obj = superobj(cfg)
obj.preprocess()
obj.process()
obj.write()
--
https://mail.python.org/mailman/listinfo/python-list

Re: The right way to use config files

[Fabien]

Hi,

On 09.08.2014 18:16, Dennis Lee Bieber wrote:

Better, in my view, is to have the import module set up default values
for everything, AND have a function at the bottom of the form

def initialize(fid=None):
if fid:
# parse file fid replacing the module level items
# this may require making a them all globals since
# assignments inside this function would be locals

And then your main program

import myconfig
...
myconfig.initialize(sys.argv[1])


Yes ok I think got it. Thanks! I like the idea and will implement it, 
this will avoid the useless superobject and allow to have to configfile 
available to anyone.


Fabien
--
https://mail.python.org/mailman/listinfo/python-list

Re: The right way to use config files

[Fabien]

On 09.08.2014 19:29, Terry Reedy wrote:

If possible, functions should *return* their results, or yield their
results in chunks (as generators). Let the driver function decide where
to put results.  Aside from separating concerns, this makes testing much
easier.


I see. But then this is also true for parameters, right? And yet we 
return to my original question ;-)



Let's say my configfile looks like this:

-
### app/config.cfg
# General params
output_dir = '..'
input_file = '..'

# Func 1 params
[func1]
enable = True
threshold = 0.1
maxite = 1
-

And I have a myconfig module which looks like:

-
### app/myconfig.py

import ConfigObj

parser = obj() # parser will be instanciated by initialize

def initialize(cfgfile=None):
   global parser
   parser = ConfigObj(cfgfile, file_error=True)
-

My main program could look like this:

-
### app/mainprogram_1.py

import myconfig

def func1():
# the params are in the cfg
threshold = myconfig.parser['func1'].as_float('threshold')
maxite = myconfig.parser['func1'].as_long('maxite')

# dummy operations
score = 100.
ite = 1
while (score  threshold) and (ite  maxite):
score /= 10
ite += 1

# dummy return
return score

def main():
myconfig.initialize(sys.argv[1])

if myconfig.parser['func1'].as_bool('enable'):
results = func1()

if __name__ == '__main__':
main()
-

Or like this:

-
### app/mainprogram_2.py

import myconfig

def func1(threshold=None, maxite=None):
# dummy operations
score = 100.
ite = 1
while (score  threshold) and (ite  maxite):
score /= 10
ite += 1

# dummy return
return score

def main():
myconfig.initialize(sys.argv[1])

if myconfig.parser['func1'].as_bool('enable'):
# the params are in the cfg
threshold = myconfig.parser['func1'].as_float('threshold')
maxite = myconfig.parser['func1'].as_long('maxite')
results = func1(threshold=threshold, maxite=maxite)

if __name__ == '__main__':
main()
-

In this case, program2 is easier to test/understand, but if the 
parameters become numerous it could be a pain...


As always, I guess I'l have to decide on a case by case basis what is best.







--
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3 is killing Python

[Fabien]

On 17.07.2014 06:47, Rick Johnson wrote: Even though i will freely 
admit that Python is the easiest

 language to learn (IMHO)

For non-informatic students (i.e the vast majority of 
science/engineering students) I don't think that's true. Less general 
languages like Matlab appear much easier to me: unified doc, unified 
IDE, unified debugger, you can spend years without being confronted to 
what an object is, etc.


 You and i don't use print, and especially not input all
 that much, but both of these (types of) functions are
 *VITAL*  lifelines for the noob when learning a language!

Some argue that making print() working like all other python functions 
make it more consistent. Here for example:

http://tinyurl.com/o2oxp9m

 Not to mention the issues of looking at the wrong version
 of a tutorial when using the other version of Python.
 Again, you and i won't make these mistakes, but a noob will!

It happened to me quite often that interesting tutorials where available 
in py2 only, despite the fact that all the concerned libraries were 
ported to py3 long ago.
But on the other hand, this is not python specific. Forums keep track of 
all questions/answers and some very old threads remain highly visible in 
the search results, making new users reinvent the wheel all the time. 
Everyone should be able to decide if the information found on blogs, 
forums or even newspapers is up-to-date or not.


Fab
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python 3 is killing Python

[Fabien]

My two cents as a new pythonista and a scientist: isn't python2 killing 
python? This old stuff is everywhere in the tutorials, docs, etc. and 
this is quite annoying. When I download a python notebook, the first 
thing I have to do is to translate it to py3. Which is not an easy task, 
given the fact that I am the tutorial *user* and am supposed to learn ;)


Side note: Debian/Ubuntu/Mint announced the total migration to python3, 
which is good but still work in progress:

https://wiki.ubuntu.com/Python/3
--
https://mail.python.org/mailman/listinfo/python-list

Error in PyDev but not in the standard python interpreter

[Fabien]

Hi,

this is probably not the best place for this topic but I know this forum 
is widely read. I take this opportunity and apology at the same time.


If I run this in my python3 interpreter everything works fine:

mowglie@flappi ~ $ python3
Python 3.3.2+ (default, Feb 28 2014, 00:52:16)
[GCC 4.8.1] on linux
Type help, copyright, credits or license for more information.
 from netCDF4 import num2date
 from datetime import datetime
 d1 = num2date(0, 'days since 1000-01-01', 'standard')
 d2 = datetime(2000, 1, 1)
 print(d1  d2)
True

But if I run the code in PyDev I get:
print(d1  d2)
TypeError: unorderable types: datetime()  datetime.datetime()

When debugging in pyDev, everything looks fine:
 sys.version
Out[28]: '3.3.2+ (default, Feb 28 2014, 00:52:16) \n[GCC 4.8.1]'
 ?d1
Type:   datetime
String Form:1000-01-01 00:00:00
File:   /usr/local/lib/python3.3/dist-packages/netcdftime.py
Docstring:
[...]
 ?d2
Type:   datetime
String Form:2000-01-01 00:00:00
File:   /usr/lib/python3.3/datetime.py
Docstring:
[...]
 d1  d2
Traceback (most recent call last):
  File 
/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py, line 
2732, in run_code

exec(code_obj, self.user_global_ns, self.user_ns)
  File ipython-input-29-415dec1be9aa, line 1, in module
d1  d2
TypeError: unorderable types: datetime()  datetime.datetime()


So they are two instances of the same object but something in pyDev 
doesn't want to compare them. Any Hint?


Thanks!

Fabien



--
https://mail.python.org/mailman/listinfo/python-list

Re: Error in PyDev but not in the standard python interpreter

[Fabien]

Hi Chris,

thanks for the hint. Indeed they are not an instance of the same class:

 isinstance(d1, datetime)
Out[6]: False
 isinstance(d2, datetime)
Out[7]: True

so this is something I should check with the NetCDF4 package developers.

While The python interpreter can compare them, my pydev environment 
can't. Is pydev doing something better then python3 alone?


Btw, the test case is extracted from the netCDF4package test suite, 
which passes on the linux command line but not on pydev because of this 
one code block...




On 24.06.2014 15:47, Chris Angelico wrote:

On Tue, Jun 24, 2014 at 11:28 PM, Fabien fabien.mauss...@gmail.com wrote:

So they are two instances of the same object but something in pyDev doesn't
want to compare them. Any Hint?


Are they really instances of the same class? One of them comes from
/usr/local/lib/python3.3/dist-packages/netcdftime.py and the other
comes from /usr/lib/python3.3/datetime.py - so they might be virtually
identical (or they might not), but they're not the actual same class,
and when __lt__ does its isinstance check, it doesn't pass.

The real question is: Why is netCDF4 not using import datetime to
get its datetime? And I think that's best answered with this, lifted
from the netcdftime.utime docstring [1]:


The datetime instances returned by C{num2date} are 'real' python datetime
objects if the date falls in the Gregorian calendar (i.e.
C{calendar='proleptic_gregorian', 'standard'} or C{'gregorian'} and
the date is after 1582-10-15). Otherwise, they are 'phony' datetime
objects which are actually instances of C{L{netcdftime.datetime}}.  This is
because the python datetime module cannot handle the weird dates in some
calendars (such as C{'360_day'} and C{'all_leap'}) which don't exist in any real
world calendar.


(Similar info can be found in the docstring for netcdftime.datetime
itself, but without the examples.)

ISTM netcdftime.datetime should define __lt__() to permit comparisons
with either others of itself or datetime.datetime objects, but in the
absence of that, you're stuck with either ensuring that you're working
with Gregorian dates, or writing your own comparison function.

Good luck!

ChrisA

[1] 
https://code.google.com/p/netcdf4-python/source/browse/trunk/netcdftime.py?r=1117#522



--
https://mail.python.org/mailman/listinfo/python-list

Re: Error in PyDev but not in the standard python interpreter

[Fabien]

Hi Chris,

On 24.06.2014 16:04, Chris Angelico wrote:

Here's what I'd try:


import sys
sys.modules[d1.__class__.__module__].__file__
sys.modules[d2.__class__.__module__].__file__


that was it!

netCDF4 has two repositories online, one on google code and one on 
github. I mistakenly installed the old one, then I noticed the new one 
and obviously some old stuffs remained in the dist-packages directory. I 
removed them from my path and everything works fine in pydev now.


Thanks a lot!

Fab

--
https://mail.python.org/mailman/listinfo/python-list

Re: Error in PyDev but not in the standard python interpreter

[Fabien]

On 24.06.2014 16:35, Chris Angelico wrote:

Would be nice if there could be some clear indication that this is the
official and current repo.


indeed. Also, the install procedure is a bit dangerous for new python 
users like me. NetCDF4 is out since 2008, it would be great if SciPy 
could support it.


Other related annoying stuff when learning is the good old 2.7 VS 3 problem:

mowglie@flappi ~ $ apt-cache search SciPy
[...]
python-scipy - scientific tools for Python
python3-scipy - scientific tools for Python 3

mowglie@flappi ~ $ apt-cache search python basemap
[...]
python-mpltoolkits.basemap - matplotlib toolkit to plot on map

- no native python3 package for basemap - compilation required...

But I'll stick to it. If so many people say it's great for scientific 
computing, python must be great right? ;-)




--
https://mail.python.org/mailman/listinfo/python-list

Re: Python and Math

[Fabien]

Hi everyone,

I am new on this forum (I come from IDL and am starting to learn python)

This thread perfectly illustrates why Python is so scary to newcomers: 
one question, three answers: yes, no, maybe.


Python-fans sure would argue freedom of choice is the most important, 
but being able to find the right tool for me in less than three days 
surely is important too. The paradox of choice ;-)


Fabien




--
https://mail.python.org/mailman/listinfo/python-list

Re: SOAPpy and callback

[fabien . benard]

Ok guys, thanks for the lesson! I'm probably not going to try to
implement this solution. However, I've learned a lot!

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: SOAPpy and callback

[fabien . benard]

SOAP standard doesn't provide any callback, and it looks like SOAPpy
doesn't too.
I remember using callbacks with Javascript and SOAP in Web pages. I was
just wondering if there could be the same with Python.

Thanks for your answer.

-- 
http://mail.python.org/mailman/listinfo/python-list

SOAPpy and callback

[fabien . benard]

Hello,

I'm trying to find how to use a callback in a SOAP client using SOAPpy.
Does SOAPpy have to manage it, or does Python include some API to do
it?

Thanks a lot.
Fabien

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python certification

[Fabien]

 Python is not about certificates or diplomas, so do not spend any
 money on it (the other guy was only joking).

Even if Python is not about certificates, I think it's not the case for
some company. And when you say to a company that you know how to dev. in
Python, they don't know anything about your level.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: open a mail and...

[Fabien]

 Is it possible to open a mail and download its files attached in a
 hard-disk using a python script?

Use the poplib to do that. The documentation for this module can be
found at http://docs.python.org/lib/module-poplib.html.

-- 
Fabien

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: gtk/qt scintilla help !

[fabien]

Neil Hodgson wrote:
 Fabien:
 
 
There is a lexPOV.cxx file in the package, but I can not find any POV 
keywords in any file ( there are about 150 POV keywords). Did i miss it, 
and if not, how do I create one and include it for the library building ?
 
 
The Scintilla web site, documentation and mailing list are:
 http://www.scintilla.org/
 http://scintilla.sourceforge.net/ScintillaDoc.html
 http://mailman.lyra.org/mailman/listinfo/scintilla-interest
 
Keyword lists are often very large as it is frequently desired to 
 highlight all the identifiers in an API. The Win32 API is around 10 
 megabytes in SciTE .api format. 
 
The SciTE editor is shipped with some keyword lists, including some 
 for POV in the pov.properties file.
Yes I saw that  file, but as far as I know, it is used by and only by 
SciTE. I thought at first that I could use it for pygtkScintilla.

 
To set keywords, call SCI_SETKEYWORDS as described on the 
 documentation page.
That is going to be tough, I do not know anything about C. It is going 
to be easier if I could find a file with a syntax close to that of POV 
and replace of the keywords found in it with POV keywords ,and the compile.

 
I do not personally use the Qt or PygtkScintilla wrappers for 
 Scintilla.

Thanks for the reply and the links

Fabien
 
Neil
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: gtk/qt scintilla help !

[fabien]

John Ridley wrote:
 fabien wrote on Sun, 01 May 2005 05:40:05 -0700:
 
Hello John,
 
Hi,
 
 
 Hello Fabien
 
 
I am writing a POV-RAY editor with Python using
either QT or GTK as GUI 'wrapper'. ( I am still trying both )

[snip]

I have also downloaded qscintilla-1.62-gpl-1.5.1.
As for GTK, I also found the LexPOV.cpp file, with
no POV keywords in it and with no POV keywords in
any of the file in the package.
The compilation builds then move the library
libqscintilla.so to /usr/lib/qt3/lib.

How do I link the newly built library to the folling qt script :

[snip]
 
 
 QScintilla is a port to Qt of the Scintilla editor control, and PyQt
 provides the python bindings for it. Scintilla has several dozen lexers
 (including one for POV), and QScintilla currently provides class
 wrappers for a small selection of them - but not POV, as you will see
 from the documentation:
 
 http://www.river-bank.demon.co.uk/docs/qscintilla/hierarchy.html
 
 Given this, there are two routes you can go down. You could ask the
 developer of QScintilla to provide support for POV by making a request
 via the mailing list:
 
 http://www.riverbankcomputing.co.uk/pyqt/mailinglist.php
 
 Or you could write your own POV lexer class in python using the
 existing QScintilla APIs. This is not particularly difficult, but it
 does take some working out. As you already know, a list of POV keywords
 is required. You will then need to look at the source (LexPOV.cpp and
 SciLexer.h) to see how the keywords are used and how the lexical states
 tie up with the SCE_POV enum. QScintilla's lexer classes are mainly
 wrappers around this information, so once you've worked it out, writing
 a python class based on QextScintillaLexer is fairly straightforward:

Well I have tried to use two files called qextscintillalexerlua.cpp 
and qextsinctillalexerlua.h. I noticed that the LUA language used the 
same type of syntax as for POV. I copied them both to 
qextscintillalexerpov and replaced all the keywords found in them with 
POV-keywords.
I compiled and it worked : I got a libqscintilla.so that was modified in 
/usr/lib/qt3/lib.

The hitch is that a scintilla PYQT script also needs a file called
/usr/share/sip/qextscintillalexerperl.sip ( here for perl ).

I do not have a clue how to build a qextscintillalexerpov.sip file.

I looked at: http://www.river-bank.demon.co.uk/docs/sip/sipref.html
but it is all chinese to me. A do not  know C.


- This is where I am going to contact river-bank for information.



 
 
class LexerPOV(QextScintillaLexer):
def __init__(self, parent, name):
QextScintillaLexer.__init__(self, parent, name)
def lexer(self):
return pov
def color(self, style):
# styles map to (SCE_POV enum)
# return your own QColor
# or return the base class default
return QextScintillaLexer.color(self, style)
def keywords(self, set):
# if set == 0:
# return pov keywords
# elif ...
return 0
def description(self, style):
# if style == 0:
# return self.tr(Default)
# elif ...
return QString.null
 
 
Thanks for the example and for the information

Fabien

to reply, remove '_nospam_'

 Of course, this is a minimal lexer class - it is possible to be a lot
 more sophisticated than what is suggested here.
 
 
 John Ridley
 
 Send instant messages to your online friends http://uk.messenger.yahoo.com 
-- 
http://mail.python.org/mailman/listinfo/python-list

gtk/qt scintilla help !

[fabien]

Hi,
I am writing a POV-RAY editor with Python using either QT or GTK as GUI 
'wrapper'. ( I am still trying both )

* * * * PYGTK * * * *
I have downloaded PygtkScintilla-1.99.5.
There is a lexPOV.cxx file in the package, but I can not find any POV 
keywords in any file ( there are about 150 POV keywords). Did i miss it, 
and if not, how do I create one and include it for the library building ?

Anyway, I compiled pyscintilla and got a library file called 'scintilla.a'.
Now how do I link the scintilla.a library to this GTK script example :
8--- snip 8
import pygtk
pygtk.require('2.0')
import gtk
import scintilla
class GUI:
def __init__(self):
window = gtk.Window(gtk.WINDOW_TOPLEVEL)
window.set_title(Pyvon)
window.connect(destroy, lambda w:gtk.main_quit())
table = gtk.Table(10,10)
# Here we connect notifications as they are gtk signals as well
editor = scintilla.Scintilla()
#s.connect(CharAdded, char_added_notif)
window.add(table)
table.add(editor)
editor.show()
window.set_size_request(200, 300)
window.show_all()
def main():
gtk.main()
return 0
if __name__ == __main__:
GUI()
main()
8--- snip 8


The same goes for QT :

* * * * PYQT * * * *
I have also downloaded qscintilla-1.62-gpl-1.5.1.
As for GTK, I also found the LexPOV.cpp file, with no POV keywords in it 
and with no POV keywords in any of the file in the package.
The compilation builds then move the library libqscintilla.so to 
/usr/lib/qt3/lib.

How do I link the newly built library to the folling qt script :
8--- snip 8
import sys, string
from qt import *
from qtext import *
fileopen = [
'16 13 5 1',
'. c #040404',
'# c #808304',
'a c None',
'b c #f3f704',
'c c #f3f7f3',
'a...',
'.aaa.a.a',
'a..a',
'a......a',
'.bcb...a',
'.cbcbcbcbc.a',
'.bcbcbcbcb.a',
'.cbcb...',
'.bcb.#.a',
'.cb.#.aa',
'.b.#.aaa',
'..#.',
'...a'
]
filesave = [
'14 14 4 1',
'. c #040404',
'# c #808304',
'a c #bfc2bf',
'b c None',
'..',
'.#..a.',
'.#....',
'.#..#.',
'.#..#.',
'.#..#.',
'.#..#.',
'.####.',
'..',
'.##.#.',
'.##..aa.#.',
'.##..aa.#.',
'.##..aa.#.',
'b.'
]
fileprint = [
'16 14 6 1',
'. c #00',
'# c #848284',
'a c #c6c3c6',
'b c #00',
'c c #ff',
'd c None',
'd.dd',
'..dd',
'.c.c.ddd',
'ddd..ddd',
'ddd.c.cd',
'dd..a.a.',
'd..a.a..',
'.aa.a.a.',
'.aa.',
'.aa###aa.a.d',
'.aabbbaa...d',
'.a.d',
'd.a.a.dd',
'dd...ddd'
]
fileOpenText = \
'''img source=fileopen
Click this button to open a emnew file/em.brbr
You can also select the bOpen/b command from the bFile/b menu.'''
fileSaveText = \
'''Click this button to save the file you are editing.brbr
You will be prompted for a filename.brbr
You can also select the bSave/b command from the bFile/b menu.'''
filePrintText = \
'''Click this button to print the file you are editing.brbr
You can also select the bPrint/b command from the bFile/b menu.'''
editorList = []
class QSci(QextScintilla):
def __init__(self,parent):
QextScintilla.__init__(self,parent)
self.setUtf8(1)
#lex = QextScintillaLexerPython(self)
#QextScintillaLexerCPP
#lex = QextScintillaLexerCPP(self)
#lex = QextScintillaLexerPov(self)
lex = QextScintillalexerCSharp(self)
self.setLexer(lex)
self.setBraceMatching(QextScintilla.SloppyBraceMatch)
self.setAutoIndent(1)
self.setIndentationWidth(4)
self.setIndentationGuides(1)
self.setIndentationsUseTabs(0)
self.setAutoCompletionThreshold(1)

class ApplicationWindow(QMainWindow):
def __init__(self):
QMainWindow.__init__(self,None,'example application main 
window',Qt.WDestructiveClose)

self.filename = QString.null
self.printer = QPrinter()
self.fileTools = QToolBar(self,'file operations')
openIcon = QPixmap(fileopen)
self.fileOpen = QToolButton(QIconSet(openIcon),'Open 
File',QString.null,self.load,self.fileTools,'open file')

saveIcon = QPixmap(filesave)
self.fileSave = QToolButton(QIconSet(saveIcon),'Save 
File',QString.null,self.save,self.fileTools,'save file')

printIcon = QPixmap(fileprint)
self.filePrint = QToolButton(QIconSet(printIcon),'Print 
File',QString.null,self.printDoc,self.fileTools,'print file')

QWhatsThis.whatsThisButton(self.fileTools)