[Python-Dev] Re: Jump on C by PyEval_SetTrace Python 3.7.7 - SOLUTION

2020-03-24 Thread Xavier de Gaye




On 3/23/20 11:17 PM, Leandro Müller wrote:

Hello Victor Stinner.

Thanks.
Your tip helped me a lot.
I understood that I need to get position on bycode of the line.
So I create the funcion to get position inside of the bycode.
Now I can to jump the correct position on trace.

frame->f_lasti = checkBycodePosition(frame->f_code, 11);

There are cases where it is invalid to change f_lineno to avoid a python stack 
corruption.
See the comments of the f_lineno setter frame_setlineno() in 
Objects/frameobject.c, the comments at the start of the function and inside the 
code itself.

Xavier
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J55XW45WYEI4F5BUX43OQGPDC7S22MCO/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Why doesn't venv also install python3*-config?

2020-01-09 Thread Xavier de Gaye




On 1/9/20 3:18 PM, Victor Stinner wrote:

Is the "ARM64" python-config (shell script) executed on the x86-64 host?


A cross-compilation means that there is probably no build framework on the 
target platform and therefore the build configuration of the cross-compilation 
of Python is not very useful there.


Which build command rely on python-config? Is it to cross-compile 3rd
party C extensions on the host?


It is useful when cross-compiling an application that embeds Python.
And I have a hunch that this may be also useful to fix the cross-compilation of 
third-party extension modules :)

Xavier
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2DRE2EEMJ4IYNC52ZV6HA4FNNIK5PRL4/
Code of Conduct: http://python.org/psf/codeofconduct/


[Python-Dev] Re: Why doesn't venv also install python3*-config?

2020-01-09 Thread Xavier de Gaye

On 1/8/20 5:53 PM, Victor Stinner wrote:


You may get the wrong information if you pick the wrong python-config script :-(

IMHO we should add a new module (problem: how should it be called? pyconfig?)


The shell script python-config has been introduced by bpo issue 16235 named "Add 
python-config.sh for use during cross compilation" in order "to behave exactly the same 
as python-config.py except it doesn't depend on a Python interpreter" as stated in the OP. So 
it does not make sense to say that one may pick the wrong python-config script. Its purpose is to 
provide cross-compilation configuration data independently of any Python interpreter running 
natively on the build machine used to do the cross-compilation.

FWIW the shell script python-config is generated from Misc/python-config.sh by 
configure and updated by the Makefile.

Xavier
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UNIJPEYGZHDW2UGBK4JNI5TEDY3MDDPF/
Code of Conduct: http://python.org/psf/codeofconduct/


Re: [Python-Dev] PEP 558: Defined semantics for locals()

2019-05-30 Thread Xavier de Gaye

Currently f_locals is documented as readonly [1].

The PEP says:

* "Don't change what isn't broken": the current tracing mode problems are caused
  by a requirement that's specific to tracing mode (support for external
  rebinding of function local variable references), so it made sense to also
  restrict any related fixes to tracing mode

However, actually attempting to implement and document that dynamic approach
highlighted the fact that it makes for a really subtle runtime state dependent
behaviour distinction in how ``frame.f_locals`` works, and creates several
new edge cases around how ``f_locals`` behaves as trace functions are added
and removed.

Accordingly, the design was switched to the current one, where
``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always
a dynamic snapshot, which is both simpler to implement and easier to explain.


Do these edge cases still exist when f_locals write access is restricted to 
code executed by the tracing function (which is more restrictive than 'tracing 
mode') ?
We can use the condition frame->f_trace not NULL and tstate->tracing true 
(tstate being a pointer to the PyThreadState structure) to know when code is executed 
by the tracing function [2]:
* The condition on tstate->tracing allows to figure out if we are running a 
frame executed by the trace function as opposed to a frame that is being traced or 
a frame executed in 'regular operation'.
* The condition on frame->f_trace removes the ambiguity whether tstate->tracing 
is set by a tracing function or by a profiling function.

[1] In section 'Frame objects' at 
https://github.com/python/cpython/blob/master/Doc/reference/datamodel.rst#the-standard-type-hierarchy
[2] Except that frame->f_trace is NULL in a 'PyTrace_CALL' trace event, so 
f_locals would remain readonly in that case. But this event is always followed by 
a 'PyTrace_LINE' event anyway so this limitation is not important IMO.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] duplicate method names in tests

2019-04-13 Thread Xavier de Gaye
The last post [1] in issue bpo-16079 lists the methods in Lib/test
that have duplicate names and that should be fixed.

Xavier

[1] https://bugs.python.org/issue16079#msg340168
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] bedevere pipelines hang on github

2019-04-08 Thread Xavier de Gaye
Thanks.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] bedevere pipelines hang on github

2019-04-07 Thread Xavier de Gaye
bedevere/issue-number and bedevere/news are not triggered for some
reason at https://github.com/python/cpython/pull/12708 and hang
forever with "Expected — Waiting for status to be reported ".

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] buildbottest on Android emulator with docker

2019-02-20 Thread Xavier de Gaye

Timeout (360 seconds) reached; failed to start emulator
---> Device ready.
---> Install Python on the emulator.
/home/pydev/build/python-native/python -B 
/home/pydev/abifa/Android/tools/install.py
error: device 'emulator-5556' not found


I can reproduce this error after removing the kvm kernel modules (I am on 
ArchLinux and ArchLinux kernels provide the required kernel modules to support 
KVM virtualization).  So my fault, forgot to explain that kvm is required to 
run this docker image, kvm is not required though when the docker image is 
built for other architectures (non x86_64), but it is very very slow then.

To fix the problem one must install the Linux distribution packages that 
provide KVM virtualization.  Whatever the Linux distribution it may also be 
necessary to enable virtualization support in the BIOS.

I will enter an issue and update the documentation at 
https://gitlab.com/xdegaye/abifa.

Thanks Stephane for the report.

Xavier


On 2/19/19 4:52 PM, Stephane Wirtel wrote:

Hi Xavier,

I get this exception

Timeout (360 seconds) reached; failed to start emulator
---> Device ready.
---> Install Python on the emulator.
/home/pydev/build/python-native/python -B 
/home/pydev/abifa/Android/tools/install.py
error: device 'emulator-5556' not found
unzip -q /home/pydev/dist/python3.8-android-24-x86_64-stdlib.zip -d 
/tmp/tmpt_v_gdbo
/home/pydev/android/android-sdk/platform-tools/adb -s emulator-5556 shell mkdir 
-p /data/local/tmp/python
Command "('/home/pydev/android/android-sdk/platform-tools/adb', '-s', 
'emulator-5556', 'shell', 'mkdir -p /data/local/tmp/python')" returned non-zero exit 
status 1
/home/pydev/abifa/Android/emulator.mk:57: recipe for target '_install' failed
make: *** [_install] Error 1

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Is distutils.util.get_platform() the "current" or the "target" platform

2019-02-19 Thread Xavier de Gaye

[Any reason for dropping python-dev?]


Sorry. just clicked the wrong button.


And the answer is a resounding "yes, it returns the target platform"? It seems 
you're saying this, but the wording of your email sounds just enough of a question that 
I'm not sure whether you are definitively answering it or not.


The answer is yes indeed, it returns the target platform.
This is a listing of the non-obvious steps that lead to this conclusion.

Xavier


On 2/19/19 8:45 PM, Steve Dower wrote:

[Any reason for dropping python-dev?]

On 19Feb2019 1139, Xavier de Gaye wrote:

Is distutils.util.get_platform() the "current" or the "target" platform ?


I *think* you're answering this below, yes?


* When cross-compiling on posix platforms using autoconf, configure is
   run with the '--host=host-type' [1] command line option to specify
   the target platform.

* The AC_CANONICAL_HOST macro is used by configure.ac to get the
   canonical variables `host' and `host_cpu' [2].
   Those variables are used to compute _PYTHON_HOST_PLATFORM.

* The Makefile generated by configure runs setup.py,
   generate-posix-vars, etc... on the build platform using
   PYTHON_FOR_BUILD, a native python interpreter that is set
   to run with the _PYTHON_HOST_PLATFORM environment variable.

* get_platform() in setup.py and in Lib/distutils/util.py returns the
   value of _PYTHON_HOST_PLATFORM when cross-compiling.

So the process of cross-compilation on posix platforms has
get_platform() return the target ('host' in autoconf terminology) platform.

[1] 
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Specifying-Target-Triplets.html
[2] 
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Canonicalizing.html


And the answer is a resounding "yes, it returns the target platform"? It seems 
you're saying this, but the wording of your email sounds just enough of a question that 
I'm not sure whether you are definitively answering it or not.

Cheers,
Steve


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] buildbottest on Android emulator with docker

2019-02-19 Thread Xavier de Gaye
https://gitlab.com/xdegaye/abifa

The table of content of the first url in my initial post gives also a
link to that repository.under the name 'Repository'.

Xavier

On Tue, Feb 19, 2019 at 4:54 PM Stephane Wirtel  wrote:
>
> Hi Xavier,
>
> I get this exception
>
> Timeout (360 seconds) reached; failed to start emulator
> ---> Device ready.
> ---> Install Python on the emulator.
> /home/pydev/build/python-native/python -B 
> /home/pydev/abifa/Android/tools/install.py
> error: device 'emulator-5556' not found
> unzip -q /home/pydev/dist/python3.8-android-24-x86_64-stdlib.zip -d 
> /tmp/tmpt_v_gdbo
> /home/pydev/android/android-sdk/platform-tools/adb -s emulator-5556 shell 
> mkdir -p /data/local/tmp/python
> Command "('/home/pydev/android/android-sdk/platform-tools/adb', '-s', 
> 'emulator-5556', 'shell', 'mkdir -p /data/local/tmp/python')" returned 
> non-zero exit status 1
> /home/pydev/abifa/Android/emulator.mk:57: recipe for target '_install' failed
> make: *** [_install] Error 1
>
>
> Is there a repository with a bug tracker because I would not want to
> pullute this ML with that.
>
> Thank you,
>
> Stéphane
>
>
> --
> Stéphane Wirtel - https://wirtel.be - @matrixise
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/xdegaye%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] buildbottest on Android emulator with docker

2019-02-15 Thread Xavier de Gaye

The following command runs the buildbottest on an Android emulator with docker 
(it will use a little bit more than 11 GB):

$ docker run -it --privileged xdegaye/abifa:r14b-24-x86_64-master

This command does:
* pull an image from the Docker hub (only the first time that the command is 
run, note that this a 2 GB download !) and start a container
* pull the latest changes from the GitHub cpython repository and cross-compile 
python
* start an Android emulator and install python on it
* run the buildbottest target of cpython Makefile

The image is built from a Dockerfile [2].

This same image can also be used with the 'bash' command line argument to enter 
bash in the container and run python interactively on the emulator [1].  If the 
'docker run' command also sets a bind mount to a local cpython repository, then 
it is possible to develop/debug/fix python on the emulator running in this 
container using one's own clone of cpython.

[1] documentation at https://xdegaye.gitlab.io/abifa/docker.html
[2] Dockerfile at 
https://gitlab.com/xdegaye/abifa/blob/master/docker/Dockerfile.r14b-24-x86_64-master

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Python3 compiled listcomp can't see local var - bug or feature?

2018-06-08 Thread Xavier de Gaye

On 06/06/2018 03:51 PM, Nick Coghlan wrote:
> On 6 June 2018 at 15:31, Rob Cliffe via Python-Dev mailto:python-dev@python.org>> wrote:
> ...
> *In other words, it looks as if in Python 3.6.5, the compiled list 
comprehension**
> **can "see" a pre-existing global variable but not a local one.*
>
> Yes, this is expected behaviour - the two-namespace form of exec (which is 
what you get implicitly when you use it inside a function body) is similar to a 
class body, and hence nested functions
> (including the implicit ones created for comprehensions) can't see the top 
level local variables.


In issue 13557 [1] Amaury gives the following explanation by quoting the documentation [2] "Free variables are not resolved in the nearest enclosing namespace, but in the global namespace" and hints 
at the same solution that is proposed by Nick.


FWIW in issue 21161 [3] folks have been bitten by this when trying to run a 
list comprehension in pdb.

[1] https://bugs.python.org/issue13557
[2] 
http://docs.python.org/py3k/reference/executionmodel.html#interaction-with-dynamic-features
[3] https://bugs.python.org/issue21161

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-15 Thread Xavier de Gaye

On 12/14/2017 02:59 PM, Victor Stinner wrote:
> It seems like Android is evolving quickly, would say quicker than
> Python releases. I'm asking if it's a good idea to put a recipe aside
> the Python source code for one specific Android API version? Would it
> still make sense to build for NDK v14 in 2 or 5 years?

NDK 14 has been released in march 2017 and the latest release is NDK 16. There are sometimes major changes between releases and I think it is critical to ensure that the builds all use the same NDK 
release for that reason. Supporting another NDK release is just a substitution in one of the files of the build system and I am sure that in 2 or 5 years there would have been a core developer smart 
enough to make that substitution (this could even have been me, I will only be 71 years old in 5 years :-)). Anyway if this is a problem, this should have been discussed in a review of the PR.


There are concerns, including a concern raised by me, about supporting Android with that build system or to supporting Android at all. It has been interesting and gratifying to work on this build 
system and to get the Python test suite running on Android without failures. Given these concerns and the lack of interest in the support of Android it is time for me to switch to something else, 
maybe improve the bdb module, why not ?


Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-14 Thread Xavier de Gaye

On 12/13/2017 10:56 PM, Victor Stinner wrote:
> I looked at your scripts to build Android but I failed to use them.

You failed because you did not read the README and tried to improvise. I will change the documentation and the build process to make it simpler for those that do not have the time to RTFM :-) I have 
documented the correct steps in the PR especially for you (they are just a summary of what is in the README) after you reported that failure. So you can follow these steps any time now and use 
correctly the build system and report the results here. This would be only fair, I think.


BTW You keep saying they are scripts when the build is actually driven by 
Makefiles (with the '.mk' extension).


> Anyway, I'm not sure why these scripts have to be part of the CPython
> git repository.
>
> Technically, is there a reason to put it aside the source code and
> Unix build scripts (configure/Makefile/setup)?
> Your https://github.com/python/cpython/pull/1629 only adds new files
> without touching existing files.
>
> I suggest to create new Git project. It may be in the python
> organization, or you may start with your GitHub account.

The 'Mac' build system has its own subdirectory in the source tree and it makes 
sense as it is the reference build system for this platform. I do not see why 
this should be different for Android.


> Cross-compilation is hard, and I'm not sure that it's possible to
> build a single recipe for all Android API versions, all configuration,
> any set of libraries, etc. For Android, it seems like each developer
> might want a subtle different configuration which might not be easy to
> support.

You are mistaken, this proposal does not suggest that we are going to support "all 
Android API versions, all configuration, any set of libraries, etc.", quite the 
opposite actually.

The proposal is an Android build system for a specific API, a set of architectures using the NDK r14 toolchain and a set of optional external libraries. The build system enforces the use of NDK r14 
for example (as you have painfuly experienced). Python is tested on emulators so that there is no interference with vendor specific additions found on the Android devices or with installed PlayStore 
applications.



> Having a separated Git project would allow people to contribute more
> easily, experiment their fork, etc.
>
> What do you think?

Certainly not. We, core-devs, are very happy that no one is experimenting with 
our build system, it is complex enough as it is. The same goes for this Android 
build system.

Your suggestion seems to be driven by the failure you have experienced with this new build system and the fact that a user is also reporting a failure.  The origin of this other failure is unclear 
because I cannot reproduce it even though all the components used for the build are well defined and identical for everyone: the NDK includes the clang compiler, the libraries and the headers, the 
external libraries are downloaded by the build system, all the users use identical tools (same versions) and the same source code and the only difference may be with some utility tools such as sed, 
awk, etc...


This is a bad start for this proposal and it would have been fair to inform me that you were working in irc collaboration with this other user in testing the build system. On the other hand, these 
problems may have some positive consequences since it allows us to be aware of the fact that the bpo audience may change if we support Android and that this may be a problem. Android attracts all kind 
of developers that do not have the average expertise of unix developers and more importantly that do not have the same motivations and the same etiquette. I am now concerned by the fact that the 
quality of the bug reports on bpo may dramatically decrease if we adopt this proposal.


Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-12 Thread Xavier de Gaye

On 12/11/2017 04:14 PM, Victor Stinner wrote:
> I'm asking for precise hardware specifications since Red Hat may be
> able to provide one through the https://osci.io/ program.

Is it acceptable to run the arm buildbots only on a weekly basis ?

For all the architectures (x86_64, armv7 and arm64), the tests are run with the same Python code, built with the same tools and running on the same operating system that runs on emulators using the 
same configuration.


If we do not take into account the ctypes issue, there is only one issue (issue 26939 [1]) among all the Android issues that is specific to arm, and it is not even really an arm issue but an issue 
related to the slowness of the emulator that has been fixed by increasing in a test the switch interval with sys.setswitchinterval().


Xavier

[1] https://bugs.python.org/issue26939

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] What does Android support mean?

2017-12-11 Thread Xavier de Gaye

On 12/11/2017 03:58 PM, Carl Bordum Hansen wrote:
>
> I've been lurking at your progress with android support for about a year, and 
now that it is closing in I simply have to ask: what does it actually mean that 
android is supported? That Android apps
> will be easy to develop in Python? That I can easily run Python script or a 
REPL on my phone?

Hello Carl,

It means that we will make sure that changes made to Python to fix it or to 
improve it will not break on Android for the supported API level and 
architectures.
One can already run Python scripts or the interactive interpreter and install 
packages with pip using termux, a great application.
Kivy can be used to develop in Python apps that work on Android and you may be interested in the Python Mobile SIG [1] mailing list or in the VOC [2] transpiler that converts Python code into Java 
bytecode.


Xavier

[1] https://www.python.org/community/sigs/current/mobile-sig/
[2] https://github.com/pybee/voc
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-11 Thread Xavier de Gaye

On 12/11/2017 03:40 PM, Victor Stinner wrote:
> 2017-12-11 14:58 GMT+01:00 Xavier de Gaye <xdeg...@gmail.com>:
>> The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x
>> + maintenance version)) emulators simultaneously, so with an eight core cpu,
>> that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be
>> set to run only daily but the tests last a long time on these architectures
>> anyway.
>
> What do you mean by "maintenance version"? Do you want to add Android
> support to Python 2.7 and 3.6 as well? I would prefer to only support
> Android since the master branch.

Yes, today only the master branch, and when 3.8 is released then both 3.7 and 
3.8 would be supported. That is what I (implicitly) meant.


> Would it be possible to only run an emulator to run a test, and then
> stop it? So we could test many combo in the same host?

I may not understand your question. An emulator runs with an AVD image (Android Virtual Device) that is specific to the API and architecture (armv7, x86_64, ...), and that contains the data (file 
systems) and the configuration. The AVD image is created once and for all for each of the combinations (API, architecture). You start the emulator with the AVD that matches the type of the built 
python (API, architecture) and install that build. Multiple emulators can run simultaneously except that the build system imposes the restriction that no two identical (API, architecture) emulators 
can run concurrently except if they are not used for the same Python version (emulators are identified by their port numbers by the Android tools and the build system allocates those ports statically).


Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-11 Thread Xavier de Gaye

On 12/11/2017 12:56 PM, Victor Stinner wrote:
> 2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdeg...@gmail.com>:
>> * Given the cpu resources required to run the test suite on the arm
>> emulators,
>>it may be difficult to find a contributed buildbot worker. So it remains
>> to
>>find the hardware to run these buildbots.
>
> Do you have the hardware to host such worker? Or are you looking for a
> host somewhere? Which kind of hardware are you looking for? CPU,
> memory, network bandwidth, etc.

I cannot host the buildbots or any buildbot for that matter. I can maintain 
them.

The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x + maintenance version)) emulators simultaneously, so with an eight core cpu, that will be 6 cores running at 100%. The armv7 
and arm64 buildbot may be set to run only daily but the tests last a long time on these architectures anyway.



>> *API 24*
>>* API 21 is the first version to provide usable support for wide
>> characters
>>  and where SELinux is run in enforcing mode.
>
> Some people are looking for API 19 support. Would it be doable, or
> would it require too many changes? I know that people are running
> heavily patched Python 2.7 and 3.5 on Android with API 19.
>
> I'm not asking for a "full support" for API 19, but more if it would
> be possible to get a "best effort" level of support, like accept
> patches if someone writes them.

Not sure that python can be built on API 19. What I remember about API 19 at 
the time I started this project, is that wide characters support is not usable.
If you look at the Android version history [1] on Wikipedia referred to in my initial post, the Kit Kat (API 19) share is 16 % now and will probably be 8 % next year. Another point to consider is that 
working on a change specific to Android is tedious: the test case must be ok on the build platform and on the emulator. The emulator must be started and an installation made from scratch, and after 
few file modifications on the emulator there is no 'git status' command to tell exactly what change you are running and you must re-install from scratch. Is there a way to browse these patches to get 
a better idea of the changes involved ?


Xavier


[1] https://en.wikipedia.org/wiki/Android_version_history
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-11 Thread Xavier de Gaye

On 12/10/2017 11:07 PM, Ned Deily wrote:
> On the one hand, it would be nice to have.  But on the other, it does add a large non-zero burden to all core developers and to the release teams, to the minimum extent of trying to make sure that 
all ongoing changes don't break platform support.


Yes this platform has few quirks especially when SELinux is involved and some 
tests are harder to write because of that :-(

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-11 Thread Xavier de Gaye

On 12/10/2017 10:27 PM, Skip Montanaro wrote:
> I'm not familiar with software development on/for Android, but
> wouldn't official support also involve suitable package creation or
> does that just fall out for free from the build-for-emulator PR?
>
> Skip
>

The build-for-emulator PR allows building for a device but the packaging of an 
Android application into an installable APK (a zip file actually) is the 
responsability of the java application developer.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Support of the Android platform

2017-12-11 Thread Xavier de Gaye

On 12/10/2017 09:27 PM, Brett Cannon wrote:
> While the note from a technical standpoint is interest, Xavier, I don't quite 
see what needs to be done to support Android at this point. Are you simply asking 
we add Android API 24 as an official
> platform? Or permission to add your note to the Misc/ directory? Basically 
what are you wanting to see happen? :)

This is mainly a proposal written in the form of a PEP and open for discussion, not a PEP because it is not required by PEP 11 but it may become one. I guess the discussion may be about the choice of 
the API level, or whether the buildbot should run on emulators or on devices (not currently supported by the build system) or on any other subject.


What is left to be done to support Android at this point is to merge the PR that implements the build system, make the change in the buildbot buildmaster-config repo and test it on a worker and then 
find the hardware to run all the workers for these architectures and set it up.


Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] Support of the Android platform

2017-12-10 Thread Xavier de Gaye

The following note is a proposal to add the support of the Android platform.

The note is easier to read with clickable links at
https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst

Motivations
===

* Android is ubiquitous.
* This would be the first platform supported by Python that is cross-compiled,
  thanks to many contributors.
* Although the Android operating system is linux, it is different from most
  linux platforms, for example it does not use GNU libc and runs SELinux in
  enforcing mode. Therefore supporting this platform would make Python more
  robust and also would allow testing it on arm 64-bit processors.
* Python running on Android is also a handheld calculator, a successor of the
  slide rule and the `HP 41`_.

Current status
==

* The Python test suite succeeds when run on Android emulators using buildbot
  strenuous settings with the following architectures on API 24: x86, x86_64,
  armv7 and arm64.
* The `Android build system`_ is described in another section.
* The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable
  buildbots to run a given Android API and architecture on the emulators.
* The Android emulator is actually ``qemu``, so the test suites for x86 and
  x86_64 last about the same time as the test suite run natively when the
  processor of the build system is of the x86 family. The test suites for the
  arm architectures last much longer: about 8 hours for arm64 and 10 hours for
  armv7 on a four years old laptop.
* The changes that have been made to achieve this status are listed in
  `bpo-26865`_, the Android meta-issue.
* Given the cpu resources required to run the test suite on the arm emulators,
  it may be difficult to find a contributed buildbot worker. So it remains to
  find the hardware to run these buildbots.

Proposal


Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64
architectures built with NDK 14b.

*API 24*
  * API 21 is the first version to provide usable support for wide characters
and where SELinux is run in enforcing mode.

  * API 22 introduces an annoying bug on the linker that prints something like
this when python is started::

  ``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffe 
arg 0x14554``.

The `termux`_ Android terminal emulator describes this problem at the end
of its `termux-packages`_ gitlab page and has implemented a
``termux-elf-cleaner`` tool to strip the useless entries from the ELF
header of executables.

  * API 24 is the first version where the `adb`_ shell is run on the emulator
as a ``shell`` user instead of the ``root`` user previously, and the first
version that supports arm64.

*x86_64*
  It seems that no handheld device exists using that architecture. It is
  supported because the x86_64 Android emulator runs fast and therefore is a
  good candidate as a buildbot worker.

*NDK 14b*
  This release of the NDK is the first one to use `Unified headers`_ fixing
  numerous problems that had been fixed by updating the Python configure script
  until now (those changes have been reverted by now).

Android idiosyncrasies
==

* The default shell is ``/system/bin/sh``.
* The file system layout is not a traditional unix layout, there is no
  ``/tmp`` for example. Most directories have user restricted access,
  ``/sdcard`` is mounted as ``noexec`` for example.
* The (java) applications are allocated a unix user id and a subdirectory on
  ``/data/data``.
* SELinux is run in enforcing mode.
* Shared memory and semaphores are not supported.
* The default encoding is UTF-8.

Android build system


The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and
is documented by its `README`_. It provides the following features:

* To build a distribution for a device or an emulator with a given API level
  and a given architecture.
* To start the emulator and
+ install the distribution
+ start a remote interactive shell
+ or run remotely a python command
+ or run remotely the buildbottest
* Run gdb on the python process that is running on the emulator with python
  pretty-printing.

The build system adds the ``Android/`` directory and the ``configure-android``
script to the root of the Python source directory on the master branch without
modifying any other file. The build system can be installed, upgraded (i.e. the
SDK and NDK) and run remotely, through ssh for example.

The following external libraries, when they are configured in the build system,
are downloaded from the internet and cross-compiled (only once, on the first
run of the build system) before the cross-compilation of the extension modules:

* ``ncurses``
* ``readline``
* ``sqlite``
* ``libffi``
* ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and
  this step is skipped on those architectures.

The following extension modules are disabled 

Re: [Python-Dev] PyCharm debugger became 40x faster on Python 3.6 thanks to PEP 523

2017-03-27 Thread Xavier de Gaye

On 03/25/2017 08:57 PM, Terry Reedy wrote:
> On 3/25/2017 8:56 AM, Serhiy Storchaka wrote:
>> On 25.03.17 12:04, Victor Stinner wrote:
>>> 
https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/
>>>
>>>
>>> "What changed in Python 3.6 to allow this?
>>>
>>> The new frame evaluation API was introduced to CPython in PEP 523 and it
>>> allows to specify a per-interpreter function pointer to handle the
>>> evaluation of frames."
>>>
>>> Nice!
>>
>> Awesome! Any chance that pdb can utilize similar technique? Or this
>> doesn't make sense for pdb?
>
> According to the bdb.Bdb docstring, pdb implements a command-line user interface on top 
of bdb, while  bdb.Bdb "takes care of the details of the trace facility".  
idlelib.debugger similarly implements
> a GUI user interface on top of bdb.  I am sure that there are other debuggers 
that build directly or indirectly (via pdb) on bdb.  So the question is whether 
bdb can be enhanced or augmented with a
> C-coded _bdb or other new module.
>
> As I understand it, sys.settrace results in an execution break and function 
call at each point in the bytecode corresponding to the beginning of a (logical?) 
line.  This add much overhead.  In return,
> a trace-based debugger allows one to flexibly control stop and go execution 
either with preset breakpoints* or with interactive commands: step (one line), 
step into (a function frame), step over (a
> function frame), or go to next breakpoint.  The last is implemented by the 
debugger automatically stepping at each break call unless the line is in the 
existing breakpoint list.
>
> * Breakpoints can be defined either in an associated editor or with 
breakpoint commands in the debugger when execution is stopped.
>
> PEP 523 envisioned an alternate non-trace implementation of 'go to next 
breakpoint' by a debugger going "as far as to dynamically rewrite bytecode prior 
to execution to inject e.g. breakpoints in the
> bytecode."
> https://www.python.org/dev/peps/pep-0523/#debugging
>
> A debugger doing this could either eliminate the other 'go' commands 
(easiest) or implement them by either setting temporary breakpoints or temporarily 
turning tracing on.
>
> I presume it should be possible to make bdb.Bdb use bytecode breakpoints or 
add a new class with a similar API.  Then any bdb-based debugger to be modified to 
make the speedup available.


pdb-clone, an extension to pdb, gets about those same performance gains over
pdb while still using sys.settrace(). pdb-clone runs at a speed of less than
twice the speed of the interpreter when pdb runs at about 80 times the speed
of the interpreter. See some performance measurements at
https://bitbucket.org/xdegaye/pdb-clone/wiki/Performances.md

Given those results, it is not clear how one would get a boost of a factor 40
by implementing PEP 523 for the pdb debugger as pdb could already be very
close to the speed of the interpreter mostly by implementing in a C extension
module the bdb.Bdb methods that check whether the debugger should take
control.

Setting a trace function with sys.settrace() adds the following incompressible 
overhead:
* 15-20 % overhead: computed goto are not used in the ceval loop when tracing 
is active.
* The trace function receives all the PyTrace_LINE events (even when the frame
  f_trace is NULL :(). The interpreter calls _PyCode_CheckLineNumber() for
  each of these events and the processing in this function is the one that is
  costly. An optimization is done in pdb-clone that swaps the trace function
  with a profiler function whenever possible (i.e. when there is no need to
  trace the lines of the function) to avoid calling _PyCode_CheckLineNumber()
  (the profiler still gets PyTrace_C_CALL events but there is not such
  overhead with these events).  The performance gain obtained with this scheme
  is about 30%.

I think that the main point here is not whether to switch from sys.settrace()
to PEP 523, but first to implement the stop_here() bdb.Bdb method in a C
extension module.

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] attributes of Python code objects

2017-02-26 Thread Xavier de Gaye

In PR 218 [1], INADA Naoki wrote:
> I think attributes of Python's code object is implementation detail, even
> though PyPy follows.
> But it's not big problem until there are some Python implementation
> having different code implementation.

Python's code object attributes are described in the Data Model [2] section of
the documentation.  1) Is this description normative (to be followed by all the
implementations of the Python language)?  2) Is the description of co_lnotab
in [3] normative ?

Xavier

[1] https://github.com/python/cpython/pull/218
[2] https://docs.python.org/3/reference/datamodel.html
[3] https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the Android platform

2017-01-15 Thread Xavier de Gaye

On 01/14/2017 09:56 PM, Victor Stinner wrote:
> Great job! Thank you and Chi Hsuan Yen!
>
> Did you get feedback from users? Maybe from the Kivy community?
>
> Victor
>
> Le 14 janv. 2017 18:31, "Xavier de Gaye" a écrit :
>
> Only a few minor issues are left to be fixed before the support of the
> Android platform may be considered. See the current status at msg285493 
[1]
> in the Android meta-issue 26865 [2].


Yes Chi Hsuan Yen is a great contributor to this project.

AFAIK Kivy uses the CrystaX NDK [1] for Python 3 instead of the
Android NDK, mainly because the Android NDK did not support wide
character until API level 21.

Xavier

[1] https://www.crystax.net/en

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] support of the Android platform

2017-01-14 Thread Xavier de Gaye

Only a few minor issues are left to be fixed before the support of the
Android platform may be considered. See the current status at msg285493 [1]
in the Android meta-issue 26865 [2].

Xavier

[1] http://bugs.python.org/issue26865#msg285493
[2] http://bugs.python.org/issue26865
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the android platform

2016-04-26 Thread Xavier de Gaye

On 04/26/2016 07:12 PM, Stefan Krah wrote:
> Xavier de Gaye  gmail.com> writes:
>> This code, or part of it, could be used to setup a buildbot and in this case
>> there would not be any conflict between the GPL v3 license and the Python
>> license, I think. I don't see how it can be combined with Python 3.
>
> For the patches on the tracker I just went by your contributor agreement.
> I didn't check the lineage of the patches. Can I assume that either you
> are re-licensing GPL-stuff written by yourself to the PSF (which is a
> perfectly valid use case of the agreement) or rewriting from scratch?


Yes, I am re-licensing GPL code to the PSF for all the patches written by me
in the issues listed on http://bugs.python.org/issue26865#msg264310.  I have
only rewritten the patches from scratch in the following issues:

issue #26849: android does not support versioning in SONAME
  (using a switch case on ac_sys_system)
issue #26854: missing header on android for the ossaudiodev module
  (actually it's difficult to rewrite such an obvious patch)
issue #26855: add platform.android_ver() for android
  (using configparser; Chi Hsuan Yen is proposing a more complete 
approach)

Fixes for those three issues can also be found in other projects porting
python3 to android, the ones that I know of are:
  * Python 3 Android at https://github.com/yan12125/python3-android, author
Chi Hsuan Yen
  * python-for-android at https://github.com/kuri65536/python-for-android,
author shimoda dragon

I also browsed rapidly issue 23496 and could not find any overlap with my
patches.

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the android platform

2016-04-26 Thread Xavier de Gaye

On 04/26/2016 04:14 PM, Ethan Furman wrote:
> On 04/24/2016 12:20 AM, Xavier de Gaye wrote:
>
>> [1] https://bitbucket.org/xdegaye/pyona/src
>
> The license:
> ---
> This software is licensed under the GNU General Public License version 3 or 
later.
> ---
>
>
> Will combining your code with Python 3 be a problem?


This code, or part of it, could be used to setup a buildbot and in this case
there would not be any conflict between the GPL v3 license and the Python
license, I think. I don't see how it can be combined with Python 3.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the android platform

2016-04-26 Thread Xavier de Gaye

On 04/26/2016 04:02 AM, Nick Coghlan wrote:
> On 26 April 2016 at 06:25, Xavier de Gaye <xdeg...@gmail.com 
<mailto:xdeg...@gmail.com>> wrote:
>
> On 04/25/2016 04:53 PM, Stefan Krah wrote:
> > Eric Snow gmail.com <http://gmail.com>> writes:
>     >> On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye gmail.com 
<http://gmail.com>>
> > wrote:
> >>> Starting with API level 21 (Android 5.0), the build of python3 with 
the
> >>> official android toolchains (that is, without resorting to external
> >
> >> How does this relate tohttp://bugs.python.org/issue23496?
> >
> > As I understand, that issue seems abandoned and the patches are
> > (despite core devs asking otherwise) against 3.4.
> >
> >
> > If Xavier is willing to do so, I think it would be best to start over
> > with a new issue that integrates his work into 3.6.
>
> I will enter a new issue that lists all the new issues and the other 
already
> existing issues that, would have they been fixed, would have allowed a
> successfull cross-build and the same test suite results as described in my
> previous post.
>
>
> Thanks for this, Xavier!
>
> Once you have that, in addition to posting the link back here, you may also 
want to ping the Mobile SIG list: 
https://www.python.org/community/sigs/current/mobile-sig/


Issue 26865 [1] lists issues that may have to be fixed in the perspective of a
future support of the android platform.

Xavier

[1] http://bugs.python.org/issue26865

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the android platform

2016-04-25 Thread Xavier de Gaye

On 04/25/2016 04:53 PM, Stefan Krah wrote:
> Eric Snow  gmail.com> writes:
>> On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye  gmail.com>
> wrote:
>>> Starting with API level 21 (Android 5.0), the build of python3 with the
>>> official android toolchains (that is, without resorting to external
>
>> How does this relate to http://bugs.python.org/issue23496?
>
> As I understand, that issue seems abandoned and the patches are
> (despite core devs asking otherwise) against 3.4.
>
>
> If Xavier is willing to do so, I think it would be best to start over
> with a new issue that integrates his work into 3.6.


I will enter a new issue that lists all the new issues and the other already
existing issues that, would have they been fixed, would have allowed a
successfull cross-build and the same test suite results as described in my
previous post.

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the android platform

2016-04-25 Thread Xavier de Gaye

On 04/25/2016 04:36 PM, Eric Snow wrote:
> On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye <xdeg...@gmail.com> wrote:
>> Starting with API level 21 (Android 5.0), the build of python3 with the
>> official android toolchains (that is, without resorting to external
>> libraries
>> for wide character support) runs correctly.  With the set of patches
>> described
>> in the patches/Makefile file at [1], the cpython test suite runs[2] on the
>> android x86 and armv7 emulators with only few errors[3].  Those errors are
>> listed with their corresponding error messages, this may give a raw idea of
>> the effort needed to support this platform.
>
> How does this relate to http://bugs.python.org/issue23496?


The patches in issue 23496 address the native compilation of Android 4.4.2 on
an android device using a port of gcc on this device.  Some of these patches
are not needed anymore on Android 5.0 and it seems that the kbox_fix.patch is
needed because the KBOX application is used to build python in issue 23496.

The existing issues that are relevant to the android platform are, I think:
issue #26723: Add an option to skip _decimal module
issue #22747: Interpreter fails in initialize on systems where
  HAVE_LANGINFO_H is undefined
issue #16353: add function to os module for getting path to default shell
issue #20306: Lack of pw_gecos field in Android's struct passwd causes
  cross-compilation for the pwd module to fail

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] support of the android platform

2016-04-25 Thread Xavier de Gaye

On 04/24/2016 11:50 AM, Stefan Krah wrote:
> Xavier de Gaye  gmail.com> writes:
>> Starting with API level 21 (Android 5.0), the build of python3 with the
>> official android toolchains (that is, without resorting to external libraries
>> for wide character support) runs correctly.  With the set of patches 
described
>> in the patches/Makefile file at [1], the cpython test suite runs[2] on the
>> android x86 and armv7 emulators with only few errors[3].  Those errors are
>> listed with their corresponding error messages, this may give a raw idea of
>> the effort needed to support this platform.
>>
>> Xavier
>>
>> [1] https://bitbucket.org/xdegaye/pyona/src
>> [2] To reproduce these results, follow the instructions found in INSTALL
>>   at https://bitbucket.org/xdegaye/pyona/wiki/install
>> [3] https://bitbucket.org/xdegaye/pyona/wiki/testsuite
>
>
> This looks great, very clean!  As I understand the patches, the locale.h and
> langinfo.h problems are solved.  Do you think the following issues on the
> Python bug tracker could be closed?
>
>
> http://bugs.python.org/issue20305
> http://bugs.python.org/issue22747
> http://bugs.python.org/issue17905


Thanks.
A fix is still needed because Android does not HAVE_LANGINFO_H.
I have tried to answer your question directly in those issues.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] support of the android platform

2016-04-24 Thread Xavier de Gaye

Starting with API level 21 (Android 5.0), the build of python3 with the
official android toolchains (that is, without resorting to external libraries
for wide character support) runs correctly.  With the set of patches described
in the patches/Makefile file at [1], the cpython test suite runs[2] on the
android x86 and armv7 emulators with only few errors[3].  Those errors are
listed with their corresponding error messages, this may give a raw idea of
the effort needed to support this platform.

Xavier

[1] https://bitbucket.org/xdegaye/pyona/src
[2] To reproduce these results, follow the instructions found in INSTALL
at https://bitbucket.org/xdegaye/pyona/wiki/install
[3] https://bitbucket.org/xdegaye/pyona/wiki/testsuite

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Bug in build system for cross-platform builds

2016-03-14 Thread Xavier de Gaye

On 03/14/2016 05:34 PM, Xavier de Gaye wrote:
> Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect
> uses of recursive make.  The changeset added executable binaries as
> prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)).
> This broke cross-compilation:
> * the executables do not exist and must be cross-compiled
> * then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since
>the prerequisites are newer
> * but the executables cannot run on the build system
>
> Actually the files need not be re-generated as their timestamps have been
> setup for that purpose with 'make touch'. So a solution to the problem
> introduced by this changeset when cross-compiling could be to remove the
> binaries as prerequisites of these rules and include the recipe of their
> corresponding rules, the one used to build the executable, into the recipes of
> the original rule.  Also IMHO the Programs/_freeze_importlib.c can be used
> instead of Programs/_freeze_importlib.o as a prerequisite in the
> Python/importlib.h rule.
>
> [1] https://hg.python.org/cpython/rev/c2a53aa27cad/
> [2] http://bugs.python.org/issue22359


The pgen dependencies are lost when following my previous suggestion, which is
wrong.  I have uploaded a patch at issue 22359 that uses a conditional to 
change the
rules, based on whether a cross-compilation is being run.

Xavier

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Bug in build system for cross-platform builds

2016-03-14 Thread Xavier de Gaye

On 03/14/2016 02:26 PM, R. David Murray wrote:
>
> The repo-timestamp problem is addressed by the 'make touch' target.
>
> And yes, checking in these platform-independent artifacts is very
> intentional: less to build, fewer external dependencies in the build
> process...you don't need to *have* python to *build* python, which you
> would have to if they were not checked in.


Changeset c2a53aa27cad [1] was commited in issue 22359 [2] to remove incorrect
uses of recursive make.  The changeset added executable binaries as
prerequisites to the existing rules (Python/importlib.h and $(GRAMMAR_H)).
This broke cross-compilation:
* the executables do not exist and must be cross-compiled
* then the Python/importlib.h or $(GRAMMAR_H) target recipes must be run since
  the prerequisites are newer
* but the executables cannot run on the build system

Actually the files need not be re-generated as their timestamps have been
setup for that purpose with 'make touch'. So a solution to the problem
introduced by this changeset when cross-compiling could be to remove the
binaries as prerequisites of these rules and include the recipe of their
corresponding rules, the one used to build the executable, into the recipes of
the original rule.  Also IMHO the Programs/_freeze_importlib.c can be used
instead of Programs/_freeze_importlib.o as a prerequisite in the
Python/importlib.h rule.

[1] https://hg.python.org/cpython/rev/c2a53aa27cad/
[2] http://bugs.python.org/issue22359

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Issues not responded to.

2015-07-31 Thread Xavier de Gaye



On 07/31/2015 06:42 AM, Zachary Ware wrote:

On Thu, Jul 30, 2015 at 8:21 PM, Brett Cannon br...@python.org wrote:

Best thing I can think of is to post the Roundup search you did to find
those 400 so thoseof us who can help can just start whittling them away. You
could also share it with core-mentorship and explain we need help evaluating
these issues with the caveat we have no idea how difficult it is to do the
evaluation.


Here's a query:

https://bugs.python.org/issue?@action=search@columns=title,id,creator,activity,actor,status@sort=activitystatus=-1,1,3,4message_count=1



This is nice, thanks.
Note that this is missing the cases where more than one message was required, 
for example to send two attachments (a script as the use case, and a patch).

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Encoding of PyFrameObject members

2015-02-07 Thread Xavier de Gaye

On 02/06/2015 11:48 PM, Francis Giraldeau wrote:
 2015-02-06 6:04 GMT-05:00 Armin Rigo:

 Hi,

 On 6 February 2015 at 08:24, Maciej Fijalkowski fij...@gmail.com 
mailto:fij...@gmail.com wrote:
  I don't think it's safe to assume f_code is properly filled by the
  time you might read it, depending a bit where you find the frame
  object. Are you sure it's not full of garbage?


 Yes, before discussing how to do the utf8 decoding, we should realize
 that it is really unsafe code starting from the line before.  From a
 signal handler you're only supposed to read data that was written to
 volatile fields.  So even PyEval_GetFrame(), which is done by
 reading the thread state's frame field, is not safe: this is not a
 volatile.  This means that the compiler is free to do crazy things
 like *first* write into this field and *then* initialize the actual
 content of the frame.  The uninitialized content may be garbage, not
 just NULLs.


 Thanks for these comments. Of course accessing frames withing a signal 
handler is racy. I confirm that code encoded in non-ascii is not accessible from 
the uft8 buffer pointer. However, a call
 to PyUnicode_AsUTF8() encodes the data and caches it in the unicode object. 
Later access returns the byte buffer without memory allocation and re-encoding.

 I think it is possible to solve both safety problems by registering a handler 
with PyPyEval_SetProfile(). On function entry, the handler will call 
PyUnicode_AsUTF8() on the required frame members to
 make sure the utf8 encoded string is available. Then, we increment the 
refcount of the frame and assign it to a thread local pointer. On function return, 
the refcount is decremented. These operations
 occurs in the normal context and they are not racy. The signal handler will 
use the thread local frame pointer instead of calling PyEval_GetFrame(). Does that 
sounds good?


You could call Py_AddPendingCall() from your signal handler and access the
frame members from the function scheduled by Py_AddPendingCall().


Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Wave module support for floating point data

2014-03-02 Thread Xavier de Gaye

On 03/02/2014 12:04 AM, Terry Reedy wrote:
 On 3/1/2014 3:25 PM, Antoine Pitrou wrote:
 On Sat, 01 Mar 2014 15:08:00 -0500
 Terry Reedy wrote:
 On 3/1/2014 2:57 PM, Sebastian Kraft wrote:
 Hi everybody,

 more than a year ago I have submitted a patch to enhance the Wave module
 with read/write support for floating point data.

 http://bugs.python.org/issue16525

 Up till now this patch has not been applied nor did I get feedback if
 anything needs to be enhanced or changed before it can be committed.
 I have never been in contact with your development process and Python
 core developers, so please tell me what I should improve...

 Please subscribe to core-mentorship list and post your question there.

 I don't understand this response. You seem to be assuming that
 Sebastian is asking for guidance,

 I am reading what he posted, which ended with please tell me what I should 
improve

 This sort of question-request is routinely posted, in much the same words, on 
core-mentorship, and routinely gets a response there.


The core-mentorship current archive is only available to the list members as 
stated at the core-mentorship web site at http://pythonmentors.com/.  Curious 
to know why.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] f_lineno accessors

2013-09-12 Thread Xavier de Gaye

In issues 7238 [1], 16482 [2], 17697 [3] and 17277 [4], the line
number may be incorrect when the global trace function has been
removed but not the frame f_trace function.
A simple test (see below) in issue 17288 [5] crashes the interpreter
when setting f_lineno in a generator from a return trace function.
All those issues are few months old. There is a patch at issue 17277
[4] that fixes the first 4 issues. There is also a patch for issue
17288 [5].


##   Setting f_lineno in a generator   ##
$ cat jump.py
def gen():
for i in range(1):
yield i
lineno = 4

for i in gen():
pass

$ python -m pdb jump.py
 /tmp/jump.py(1)module()
- def gen():
(Pdb) import sys; print(sys.version)
3.4.0a1+ (default:975d1e180689, Sep  6 2013, 09:26:12)
[GCC 4.3.2]
(Pdb) break 3
Breakpoint 1 at /tmp/jump.py:3
(Pdb) continue
 /tmp/jump.py(3)gen()
- yield i
(Pdb) step
--Return--
 /tmp/jump.py(3)gen()-0
- yield i
(Pdb) jump 2
 /tmp/jump.py(2)gen()-0
- for i in range(1):
(Pdb) continue
Segmentation fault

[1] http://bugs.python.org/issue7238
[2] http://bugs.python.org/issue16482
[3] http://bugs.python.org/issue17697
[4] http://bugs.python.org/issue17277
[5] http://bugs.python.org/issue17288

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Test the test suite?

2013-08-28 Thread Xavier de Gaye
It happens that few tests are also never run because of name conflicts.
See issue 16056.

Xavier

On Wed, Aug 28, 2013 at 1:37 PM, Victor Stinner
victor.stin...@gmail.com wrote:
 Hi,

 I just noticed that tests using @requires_freebsd_version and
 @requires_linux_version decorator from test.support are never run
 since this commit (almost 2 years ago):

 changeset:   72618:3b1859f80e6d
 user:Charles-François Natali neolo...@free.fr
 date:Mon Oct 03 19:40:37 2011 +0200
 files:   Lib/test/support.py
 description:
 Introduce support.requires_freebsd_version decorator.

 ...

  raise unittest.SkipTest(
 -Linux kernel %s or higher required, not %s
 -% (min_version_txt, version_txt))
 -return func(*args, **kw)
 -wrapper.min_version = min_version
 +%s version %s or higher required, not %s
 +% (sysname, min_version_txt, version_txt))


 I don't want to blame Charles-François, nobody saw the issue during 2 years!

 No, my question is: how can we detect that a test is never run? Do we
 need test covertage on the test suite? Or inject faults in the code to
 test the test suite? Any other idea?

 I fixed the decorators in Python 3.3 (84debb4abd50) and 3.4 (f98fd5712b0e).

 Victor
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 http://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe: 
 http://mail.python.org/mailman/options/python-dev/xdegaye%40gmail.com



-- 
Xavier

Les Chemins de Lokoti: http://lokoti.alwaysdata.net
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] Can we triple quoted string as a comment?

2013-03-28 Thread Xavier de Gaye
On Tue, Mar 26, 2013 at 2:28 PM, R. David Murray wrote:
 On Mon, 25 Mar 2013 18:16:47 -0700, Raymond Hettinger wrote:
 If you're editing with Emacs, it is really easy to reflow paragraphs
 and to insert or remove multiline comments each prefixed with #.
 But with other editors, it can be a PITA and a multiline string is
 the easiest to maintain and works well when cutting-and-pasting
 the comments from somewhere else.

 Just FYI it is also very easy in vim: gq plus whatever movement prefix
 suits the situation.

And to comment out multiple lines in vim, each prefixed with #, see
:help v_b_I and :help v_b_I_example.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] The step command of pdb is broken

2012-05-05 Thread Xavier de Gaye
On Mon, Apr 30, 2012 at 12:31 PM, Xavier de Gaye wrote:
 Issue http://bugs.python.org/issue13183 raises the point that the step
 command of pdb is broken. This issue is 6 months old. A patch and test
 case have been proposed.

Other pdb commands are also broken for the same reason (no trace
function setup in the targeted caller frame).
A new http://bugs.python.org/issue14728 has been submitted with a
proposed patch for these commands and the corresponding test cases.
The patch removes a while loop from the fast path, and that should also
provide an improvement of the performance of Pdb.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


[Python-Dev] The step command of pdb is broken

2012-04-30 Thread Xavier de Gaye
Issue http://bugs.python.org/issue13183 raises the point that the step
command of pdb is broken. This issue is 6 months old. A patch and test
case have been proposed. The 'Lifecycle of a Patch' at
http://docs.python.org/devguide/patch.html says
quote
If your patch has not received any notice from reviewers (i.e., no
comment made) after a substantial amount of time then you may email
python-dev@python.org asking for someone to take a look at your patch.
/quote
I am the author of pyclewn, a Vim front end to pdb and gdb, and I
would be grateful for any progress on this issue.

The following pdb session shows the problem when running the three
modules main.py, foo.py and bar.py. After the second step command, pdb
does not stop (as it should) at lines foo.py:5 and foo.py:6, nor does
it stop to print the return value of increment().
=
main.py
 1  import foo
 2
 3  result = foo.increment(100)
 4  print('result', result)
foo.py
 1  import bar
 2
 3  def increment(arg):
 4  v =  bar.value()
 5  result = arg + v
 6  return result
bar.py
 1  def value():
 2  return 5
=
$ python -m pdb main.py
 /path_to/main.py(1)module()
- import foo
(Pdb) import sys; sys.version
'3.3.0a2+ (default:2c27093fd11f, Apr 30 2012, 10:51:35) \n[GCC 4.3.2]'
(Pdb) break bar.py:2
Breakpoint 1 at /path_to/bar.py:2
(Pdb) continue
 /path_to/bar.py(2)value()
- return 5
(Pdb) step
--Return--
 /path_to/bar.py(2)value()-5
- return 5
(Pdb) step
 /path_to/main.py(4)module()
- print('result', result)
(Pdb)
=


Xavier
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] The step command of pdb is broken

2012-04-30 Thread Xavier de Gaye
On Mon, Apr 30, 2012 at 6:57 PM, Senthil Kumaran wrote:
 Irrespective of this - Issue13183 seems to be an easy to verify bug in
 3.2 and 3.3. I think, it would most visible if you were to use a full
 screen debugger and you will notice that the return call indicator has
 jumped to the next statement (skipping return) when returning. I
 guess, that's why Xavier (pyclewn author) noted it.  The fix seems
 fine too.

 I have just requested an additional info and this particular one could
 be fixed.


Thanks for your help on this issue Senthil.

Xavier
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com