Re: [Scons-dev] Customize packaging tool

[Daniel Holth]

It was easier to make my own! It's a python stdlib tar builder modeled off
the zip tool, with an argument to prefix a directory name to all files.
This makes sdists in place without littering the desired structure in the
project root. It will be bundled in my project.

On Sat, May 25, 2019, 05:30 Daniel Holth  wrote:

> I've made a few improvements to the enscons setup.py replacement recently.
> It uses standard scons now instead of a repackaged version. It uses scons'
> packaging tool to build sdists.
> I'd like to be able to add a format to the packaging tool at runtime
> (cleanly) and to be able to change its build directory (to ./build). Sdists
> should be pax format tar and anonymized without your username on each file,
> and shouldn't have to count on a system tar executable.
>
> I don't quite understand how the existing tool works, it might be almost
> as easy to have my own replacement?
>
> Thanks
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

[Scons-dev] Customize packaging tool

[Daniel Holth]

I've made a few improvements to the enscons setup.py replacement recently.
It uses standard scons now instead of a repackaged version. It uses scons'
packaging tool to build sdists.
I'd like to be able to add a format to the packaging tool at runtime
(cleanly) and to be able to change its build directory (to ./build). Sdists
should be pax format tar and anonymized without your username on each file,
and shouldn't have to count on a system tar executable.

I don't quite understand how the existing tool works, it might be almost as
easy to have my own replacement?

Thanks
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Hashes

[Daniel Holth]

blake2 is supposed to be very fast, faster than md5. It would probably
break the 'scons uses stdlib only' rule though. https://blake2.net/

I assume to break scons you would have to update the same filename with its
md5 collision [while keeping the timestamps the same]?

People have tried to put sha1 collisions in their git repositories as test
input only to find that git breaks. They can cause mischief.

On Thu, Oct 26, 2017 at 10:00 AM Jonathon Reinhart <
jonathon.reinh...@gmail.com> wrote:

> I believe you will never encounter an accidental MD5 collision in the way
> that SCons uses it. [1] All of the MD5 collisions being publicized are
> intentional; leveraging a chosen-prefix attack. Does SCons really care to
> address the case where someone is intentionally generating collisions? I
> imagine not.
>
> MD5 is still the fastest general-purpose hashing algorithm [2]. So I so
> reason for SCons to worry about changing hash algorithms.
>
> Jonathon Reinhart
>
> [1]: https://stackoverflow.com/a/937798/119527
> [2]: https://stackoverflow.com/a/2723941/119527
>
>
> On Thu, Oct 26, 2017 at 7:58 AM, Russel Winder 
> wrote:
>
>> I may just be out of date: is SCons using MD5 for hashing?
>>
>> Clearly SCons is not that interested in security or true persistence
>> level hashing, but given the issue of clashing might MD5 now not be
>> useful?
>>
>> --
>> Russel.
>>
>> =
>> Dr Russel Winder  t: +44 20 7585 2200   voip:
>> sip:russel.win...@ekiga.net
>> 41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@winder.org.uk
>> London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] SCons performance investigations

[Daniel Holth]

At least in pypy a json sconsign would be quite fast compared to pickle.

On Fri, Jul 21, 2017, 11:39 Andrew C. Morrow 
wrote:

>
> Hi scons-dev -
>
> The following is a revised draft of an email that I had originally
> intended to send as a follow up to
> https://pairlist4.pair.net/pipermail/scons-users/2017-June/006018.html.
> Instead, Bill Deegan and I took some time to expand on my first draft and
> add some ideas about how to address some of th e issues. We hope to migrate
> this to the wiki, but wanted to share it here first for feedback.
>
> 
>
> Performance is one of the major challenges facing SCons. When compared
> with other current options, particularly Ninja, in many cases performance
> can lag significantly. That said other options by and large lack the
> extensibility and many features of SCons.
>
> Bill Deegan (SCons project co-manager) and I have been working together to
> understand some of the issues that lead to poor SCons performance in a real
> world (and fairly modestly sized) C++ codebase. Here is a summary of some
> of our findings:
>
>
>-
>
>Python code usage: There are many places in the codebase where while
>the code is correct, performance based on cpython’s implementation can be
>improved by minor changes.
>-
>
>   Examples
>   -
>
>  Using for loops and hashes to uniquify a list. Simple change in
>  Node class yielded approximately 15% speedup for null build
>  -
>
>  Using if x.find(‘some character’) >=0 instead of is ‘some
>  character’ in x (timeit benchmark shows a 10x speed difference)
>  -
>
>   Method to address
>   -
>
>  Profile the code looking for hotspots with cprofile and
>  line_profiler. Then look for best implementations of code. (Use 
> timeit if
>  useful to compare implementations. There are examples of such in the 
> bench
>  dir (see:
>  
> https://bitbucket.org/scons/scons/src/68a8afebafbefcf88217e9e778c1845db4f81823/bench/?at=default
>  )
>  -
>
>Serial DAG traversal: SCons walks the DAG to find out of date targets
>in a serial fashion. Once it finds them, it farms the work out to other
>threads, but the DAG walk remains serial. Given the proliferation of
>multicore machines since SCons’ initial implementation, a parallel walk of
>the DAG would yield significant speedup. Likely this would require
>implementation using the multiprocessing python library (instead of
>threads), since the GIL would block real parallelism otherwise. Packages
>like Boost where there are many header files can cause large increases in
>the size of the DAG, exacerbating this issue. There are two serious
>consequences of the slow DAG walk:
>-
>
>   Incremental rebuilds in large projects. Typical developer workflow
>   is to edit a file, rebuild, test. In our modestly sized codebase, we see
>   the incremental time to do an ‘all’ rebuild for a one file change can 
> reach
>   well over a minute. This time is completely dominated by the serial
>   dependency walk.
>   -
>
>   Inability to saturate distributed build clusters. In a
>   distcc/icecream build, the serial DAG walk is slow enough that not 
> enough
>   jobs can be farmed out in parallel to saturate even a modest (400 cpu)
>   build cluster. In our example, using ninja to drive a distributed full
>   build results in an approximately 15x speedup, but SCons can only 
> achieve a
>   2x speedup.
>   -
>
>   Method to address:
>   -
>
>  Investigate changing tree walk to generator
>  -
>
>  Investigate implementing tree walk using multiprocessing library
>  -
>
>The dependency graph is the python object graph: The target dependency
>DAG is modeled via python Node Object to Node Object linkages (e.g. a list
>of child nodes held in a node). As a result, the only way to determine
>up-to-date-ness is by deeply nested method calls that repeatedly traverse
>the Python object graph. An attempt is made to mitigate this by memoizing
>state at the leaves (e.g. to cache the result of stat calls), but this
>still results in a large number of python function invocations for even the
>simplest state checks, where a result is already known. Similarly, the lack
>of global visibility precludes using externally provided change information
>to bypass scans.
>-
>
>   See above re generator
>   -
>
>   Investigate modeling state separately from the python Node graph
>   via some sort of centralized scoreboarding mechanism, it seems likely 
> that
>   both the function call overhead could be eliminated and that local
>   knowledge could be propagated globally more effectively.
>   -
>
>CacheDir: There are some issues listed below. End-to-end caching
>functionality 

Re: [Scons-dev] Python 2 and Python 3 on same project

[Daniel Holth]

It would make sense to replace the file because it should be very unusual
to run the same SConstruct with multiple Pythons (unless you are a SCons
developer unfortunately) and because the extension should depend on the
version of the pickle protocol used, not the major version of Python.

On Thu, Jul 13, 2017, 16:09 Bill Deegan <b...@baddogconsulting.com> wrote:

> The previous code (which is currently commented out), had different file
> extension for py3, so there'd be no collision.
> Though a warning that a sconsign from opposite python version is present
> might be wise.
>
>
> On Thu, Jul 13, 2017 at 3:24 PM, Daniel Holth <dho...@gmail.com> wrote:
>
>> Would it make more sense to present a warning and then replace the
>> .dblite file?
>>
>> On Thu, Jul 13, 2017 at 2:09 PM Bill Deegan <b...@baddogconsulting.com>
>> wrote:
>>
>>> dblite.py line 43.
>>>
>>> If you uncomment that and then run the tests py2 and py3 some will break
>>> because they are  looking for the file name without the .py3 extension.
>>> Perhaps add a method or constant to QMTest/TestSCons.py which the
>>> default file name and use that in the tests which fail?
>>>
>>> -Bill
>>>
>>> On Thu, Jul 13, 2017 at 10:01 AM, Russel Winder <rus...@winder.org.uk>
>>> wrote:
>>>
>>>> On Thu, 2017-07-13 at 09:10 -0400, Bill Deegan wrote:
>>>> > Yes.
>>>> >
>>>> > There was a separate .py3 and .py2 version of sconsign.
>>>> > Probably time to turn that back on and resolve test failures due to
>>>> > changed
>>>> > sconsign name.
>>>> >
>>>> > Care to give it a try?
>>>> > ;)
>>>> >
>>>>
>>>> Yes, I can do that. What change do I need to make for it to work?
>>>>
>>>> --
>>>> Russel.
>>>>
>>>> =
>>>> Dr Russel Winder t:+44 20 7585 2200   voip:sip:
>>>> russel.win...@ekiga.net
>>>> 41 Buckmaster Road   m:+44 7770 465 077   xmpp:rus...@winder.org.uk
>>>> London SW11 1EN, UK  w: www.russel.org.uk skype:russel_winder
>>>>
>>>> ___
>>>> Scons-dev mailing list
>>>> Scons-dev@scons.org
>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>
>>>>
>>> ___
>>> Scons-dev mailing list
>>> Scons-dev@scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Python 2 and Python 3 on same project

[Daniel Holth]

Would it make more sense to present a warning and then replace the .dblite
file?

On Thu, Jul 13, 2017 at 2:09 PM Bill Deegan 
wrote:

> dblite.py line 43.
>
> If you uncomment that and then run the tests py2 and py3 some will break
> because they are  looking for the file name without the .py3 extension.
> Perhaps add a method or constant to QMTest/TestSCons.py which the default
> file name and use that in the tests which fail?
>
> -Bill
>
> On Thu, Jul 13, 2017 at 10:01 AM, Russel Winder 
> wrote:
>
>> On Thu, 2017-07-13 at 09:10 -0400, Bill Deegan wrote:
>> > Yes.
>> >
>> > There was a separate .py3 and .py2 version of sconsign.
>> > Probably time to turn that back on and resolve test failures due to
>> > changed
>> > sconsign name.
>> >
>> > Care to give it a try?
>> > ;)
>> >
>>
>> Yes, I can do that. What change do I need to make for it to work?
>>
>> --
>> Russel.
>>
>> =
>> Dr Russel Winder t:+44 20 7585 2200   voip:sip:
>> russel.win...@ekiga.net
>> 41 Buckmaster Road   m:+44 7770 465 077   xmpp:rus...@winder.org.uk
>> London SW11 1EN, UK  w: www.russel.org.uk skype:russel_winder
>>
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

[Scons-dev] method for calling SCons as a library?

[Daniel Holth]

My SCons-powerd Python packaging tool needs to report the name of the file
it built from an API. Here's the method I've come up with. Is there a
better way to figure out what was actually built?

0. Update sys.argv[] to pass desired arguments to SCons
1. Call SCons.Script.Main.main()
2. Catch SystemExit
3. Examine SCons.Script.DEFAULT_TARGETS
4. return interesting target name
5. Exit Python

Thanks.
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] status of pip install scons

[Daniel Holth]

My project is still https://pypi.python.org/pypi/import-scons packaged with
https://bitbucket.org/dholth/scons-wheel/src , no progress has been made
lately. To build, SCons' own built .py files need to be put into
subdirectories of scons-wheel. It uses wheel tags to put different source
code in the Python 2 & 3 versions, the Py3 version is from the github fork.

On Tue, Nov 29, 2016 at 11:08 PM Bill Deegan 
wrote:

Jason,

Feel free to chip in here.
Status is as you state.
pip in virtualenv seems to work, outside of virtualenv I don't think it's
working.

No current work on it.

-Bill

On Tue, Nov 29, 2016 at 6:10 PM, Jason Kenny  wrote:

Hi Guys,



Getting some time to get back to Parts and SCons…

Just want to check on the status of setup.py? for pip install of SCons.



I noticed that I cannot install on some linux and windows system with pip,
but can generally install with a virtualenv everywhere. I know we had a the
pip expert himself look at this a little as he was using SCons to make a
possible pip replacement. I started looking at fixing up the setup.py
myself in SCons, but wanted to check to see if there was anything else
going on here first…



Thanks

Jason

___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev


___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Odd sconsdb related test failure

[Daniel Holth]

A quick fix would be to change L124 to self._pickle_dump(self._dict, f,
PICKLE_PROTOCOL or -1)

My suspicion is that dblite's __del__ method is being called during
interpreter teardown which would explain why the module-level constant
PICKLE_PROTOCOL is None and why pdb doesn't work here, and then it decides
to *write itself to disk* inside __del__().

Actually on further reading I see dblite expects this, and keeps copies of
various constants in its own class. It could do the same with
PICKLE_PROTOCOL.

A less quick fix would be to figure out whether dblite could be made into a
context manager or something and not rely on persistence during interpreter
cleanup.

Daniel

On Mon, Sep 19, 2016 at 7:57 PM William Blevins <wblevins...@gmail.com>
wrote:

> Daniel,
>
> Apt (Debian package manager) shows 2.7.11-2. Python version shows 2.7.12+.
> I assume this happens on stable versions also.
>
> V/R,
> William
>
> On Mon, Sep 19, 2016 at 6:38 PM, Daniel Holth <dho...@gmail.com> wrote:
>
>> Is the requested pickle protocol something other than an int here? Del
>> can call sync. Just regular python 2.7?
>> https://bitbucket.org/scons/scons/pull-requests/348/centralize-the-preferred-pickle-protocol/diff#Lsrc/engine/SCons/dblite.pyT124
>>
>> On Sat, Sep 17, 2016, 22:58 William Blevins <wblevins...@gmail.com>
>> wrote:
>>
>>> Not sure if Daniel is on the dev-list, so I'll start a direct chat and
>>> see if he has any thoughts.
>>>
>>> On Sat, Sep 17, 2016 at 8:23 PM, James Corey <jc-sc...@neniam.net>
>>> wrote:
>>>
>>>> Yes, that test also fails for me, on debian and fedora.
>>>>
>>>>
>>>> On Sat, Sep 17, 2016 at 3:14 PM, William Blevins <wblevins...@gmail.com>
>>>> wrote:
>>>> > It appears that the pickle patch from earlier this week is causing a
>>>> test
>>>> > failure.
>>>> >
>>>> >
>>>> https://bitbucket.org/scons/scons/pull-requests/348/centralize-the-preferred-pickle-protocol/diff
>>>> >
>>>> >> 1/1 (100.00%) /usr/bin/python -tt test/Interactive/variant_dir.py
>>>> >> STDOUT
>>>> >>
>>>> =
>>>> >> scons: Entering directory `/tmp/testcmd.2983.M_iWAJ/work'
>>>> >> scons>>> scons: building associated VariantDir targets: build
>>>> >> gcc -o build/hello.o -c sub1/hello.c
>>>> >> gcc -o build/hello build/hello.o
>>>> >> scons: `sub1' is up to date.
>>>> >> scons>>> Touch("/tmp/testcmd.2983.M_iWAJ/markers/1")
>>>> >> scons>>> scons: building associated VariantDir targets: build
>>>> >> gcc -o build/hello.o -c sub1/hello.c
>>>> >> gcc -o build/hello build/hello.o
>>>> >> scons: `sub1' is up to date.
>>>> >> scons>>> Touch("/tmp/testcmd.2983.M_iWAJ/markers/2")
>>>> >> scons>>>
>>>> >>
>>>> >> STDERR
>>>> >>
>>>> =
>>>> >> 0a1
>>>> >> > Exception TypeError: 'an integer is required' in >>> >> > dblite.__del__ of >
>>>> ignored
>>>> >> FAILED test of .../SCons/scons/src/script/scons.py
>>>> >>  at line 620 of .../SCons/scons/QMTest/TestCommon.py (_complete)
>>>> >>  from line 674 of .../SCons/scons/QMTest/TestCommon.py (finish)
>>>> >>  from line 109 of test/Interactive/variant_dir.py
>>>> >
>>>> > Anyone else seeing this issue?
>>>> >
>>>> > V/R,
>>>> > William
>>>> >
>>>> > ___
>>>> > Scons-dev mailing list
>>>> > Scons-dev@scons.org
>>>> > https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>> >
>>>> ___
>>>> Scons-dev mailing list
>>>> Scons-dev@scons.org
>>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>
>>>
>>> ___
>>> Scons-dev mailing list
>>> Scons-dev@scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Odd sconsdb related test failure

[Daniel Holth]

Is the requested pickle protocol something other than an int here? Del can
call sync. Just regular python 2.7?
https://bitbucket.org/scons/scons/pull-requests/348/centralize-the-preferred-pickle-protocol/diff#Lsrc/engine/SCons/dblite.pyT124

On Sat, Sep 17, 2016, 22:58 William Blevins  wrote:

> Not sure if Daniel is on the dev-list, so I'll start a direct chat and see
> if he has any thoughts.
>
> On Sat, Sep 17, 2016 at 8:23 PM, James Corey  wrote:
>
>> Yes, that test also fails for me, on debian and fedora.
>>
>>
>> On Sat, Sep 17, 2016 at 3:14 PM, William Blevins 
>> wrote:
>> > It appears that the pickle patch from earlier this week is causing a
>> test
>> > failure.
>> >
>> >
>> https://bitbucket.org/scons/scons/pull-requests/348/centralize-the-preferred-pickle-protocol/diff
>> >
>> >> 1/1 (100.00%) /usr/bin/python -tt test/Interactive/variant_dir.py
>> >> STDOUT
>> >>
>> =
>> >> scons: Entering directory `/tmp/testcmd.2983.M_iWAJ/work'
>> >> scons>>> scons: building associated VariantDir targets: build
>> >> gcc -o build/hello.o -c sub1/hello.c
>> >> gcc -o build/hello build/hello.o
>> >> scons: `sub1' is up to date.
>> >> scons>>> Touch("/tmp/testcmd.2983.M_iWAJ/markers/1")
>> >> scons>>> scons: building associated VariantDir targets: build
>> >> gcc -o build/hello.o -c sub1/hello.c
>> >> gcc -o build/hello build/hello.o
>> >> scons: `sub1' is up to date.
>> >> scons>>> Touch("/tmp/testcmd.2983.M_iWAJ/markers/2")
>> >> scons>>>
>> >>
>> >> STDERR
>> >>
>> =
>> >> 0a1
>> >> > Exception TypeError: 'an integer is required' in > >> > dblite.__del__ of >
>> ignored
>> >> FAILED test of .../SCons/scons/src/script/scons.py
>> >>  at line 620 of .../SCons/scons/QMTest/TestCommon.py (_complete)
>> >>  from line 674 of .../SCons/scons/QMTest/TestCommon.py (finish)
>> >>  from line 109 of test/Interactive/variant_dir.py
>> >
>> > Anyone else seeing this issue?
>> >
>> > V/R,
>> > William
>> >
>> > ___
>> > Scons-dev mailing list
>> > Scons-dev@scons.org
>> > https://pairlist2.pair.net/mailman/listinfo/scons-dev
>> >
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] bitbucket login

[Daniel Holth]

Slightly inconvenient but a password reset solved it.

On Wed, Jul 6, 2016, 23:57 Bill Deegan  wrote:

> no problem here.
> -Bill
>
> On Wed, Jul 6, 2016 at 1:25 PM, William Blevins 
> wrote:
>
>> Lovely...
>>
>> Thanks,
>> William
>>
>> On Wed, Jul 6, 2016 at 4:37 PM, Jason Kenny  wrote:
>>
>>> I am able to get in.
>>>
>>>
>>>
>>> Jason
>>>
>>>
>>>
>>> *From:* Scons-dev [mailto:scons-dev-boun...@scons.org] *On Behalf Of 
>>> *William
>>> Blevins
>>> *Sent:* Wednesday, July 6, 2016 10:27 AM
>>> *To:* SCons developer list 
>>> *Subject:* [Scons-dev] bitbucket login
>>>
>>>
>>>
>>> Krew,
>>>
>>> It appears that Atlassian is trying to intregrate accounts. Is anyone
>>> else unable to login to Bitbucket as a result?
>>>
>>> V/R,
>>>
>>> William
>>>
>>> ___
>>> Scons-dev mailing list
>>> Scons-dev@scons.org
>>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>> ___
>> Scons-dev mailing list
>> Scons-dev@scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Fwd: [Distutils] enscons, a prototype SCons-powered wheel & sdist builder

[Daniel Holth]

Jonathon,

Grab the development enscons from https://bitbucket.org/dholth/enscons .
Run the setup2toml script to pull your setup.py arguments into an
appropriate toml. Try it out. Then let me know what worked and what didn't
work.

Thanks,

Daniel

On Thu, Jun 30, 2016 at 10:07 AM Daniel Holth <dho...@gmail.com> wrote:

> Here's the SConstruct I'm using to repackage SCons itself. I've taken
> timj's Python 3 branch and SCons + __main__.py and packaged them as
> import-scons-*.whl. I can have two totally different packages for Python 2
> and 3, and the correct one is installed with 'pip install import-scons'.
> Then you can type 'python -m SCons' to run. So far timj's Python 3 port
> does everything I've asked it to do.
>
> https://bitbucket.org/dholth/scons-wheel/src
>
> On Mon, Jun 27, 2016 at 2:13 PM Bill Deegan <b...@baddogconsulting.com>
> wrote:
>
>> Forwarding from distutils mailing list.
>> Thought it might be of interest to the community.
>>
>> -Bill
>> Co-Manager SCons Project
>> -- Forwarded message --
>> From: Daniel Holth <dho...@gmail.com>
>> Date: Sun, Jun 26, 2016 at 1:45 PM
>> Subject: [Distutils] enscons, a prototype SCons-powered wheel & sdist
>> builder
>> To:
>>
>>
>> I've been working on a prototype Python packaging system powered by SCons
>> called enscons. https://bitbucket.org/dholth/enscons . It is designed to
>> be an easier way to experiment with packaging compared to hacking on
>> distutils or setuptools which are notoriously difficult to extend. Now it
>> is at the very earliest state where it might be interesting to others
>> who are less familiar with the details of pip and Python package formats.
>>
>> Enscons is able to create wheels and source distributions that can be
>> installed by pip. Source distributions built with enscons use
>> pyproject.toml from PEP 518 ( https://www.python.org/dev/peps/pep-0518/ )
>> to install enscons before setup.py is run, and since pip does not yet
>> implement PEP 518, include a setup.py shim for PEP 518. The same
>> pyproject.toml includes most of the arguments normally passed to
>> setuptools.setup(), which are used to create the metadata files needed by
>> pip.
>>
>> Enscons converts pip's setup.py arguments to SCons ones and then invokes
>> SCons, and the project's SConstruct does the rest. For a normal from-pypi
>> installation, enscons generates just enough egg_info to let pip build a
>> wheel, then pip builds and installs the wheel which has more complete
>> metadata.
>>
>> Of course SCons is more interesting for more complicated projects. I've
>> been working on building my pysdl2-cffi binding with SCons. It includes a
>> custom Python code generation step that distutils knew nothing about. It's
>> practically a one-liner to add my custom build step to SCons, and it knows
>> how to rebuild each part of the project only when its dependencies have
>> changed. I don't know how to do that with setuptools.
>>
>> It is still a very early prototype. Don't expect it to be very good. Here
>> are some of its limitations:
>>
>>- There is no error checking.
>>- The sdists can only be built inside virtualenvs; otherwise pip
>>complains that --user and --target cannot be used together.
>>- It also doesn't work if one of the pyproject.toml build
>>requirements conflicts with something that's already installed.
>>- 'pip install .' doesn't work; it still tries to invoke 'setup.py
>>install', which is not implemented.
>>- 'setup.py develop' is not implemented. Try PYTHONPATH.
>>- I am not an experienced SCons user.
>>
>> It also relies on what I'm sure will be a short-lived repackaging of
>> SCons called 'import_scons' that makes 'python -m SCons' work, and includes
>> a Python 3 version of SCons from github.com/timj/scons that seems to
>> work well enough for my purposes.
>>
>> On the plus side, it's short.
>>
>> If you're interested in implementing Python packaging in SCons, or are
>> interested in the straightforward but tedious task of copying just enough
>> distutils trickery to implement a robust Python C extension compiler in
>> SCons, then perhaps you should check out enscons.
>>
>> Here is a SConstruct for enscons itself:
>>
>> import pytoml as tomlimport ensconsmetadata = 
>> dict(toml.load(open('pyproject.toml')))['tool']['enscons']
>> env = Environment(tools=['default', 'packaging', enscons.generate],  
>> PACKAGE_METADATA=metadata,  
>> WHEEL_TAG='py2

Re: [Scons-dev] Fwd: [Distutils] enscons, a prototype SCons-powered wheel & sdist builder

[Daniel Holth]

Here's the SConstruct I'm using to repackage SCons itself. I've taken
timj's Python 3 branch and SCons + __main__.py and packaged them as
import-scons-*.whl. I can have two totally different packages for Python 2
and 3, and the correct one is installed with 'pip install import-scons'.
Then you can type 'python -m SCons' to run. So far timj's Python 3 port
does everything I've asked it to do.

https://bitbucket.org/dholth/scons-wheel/src

On Mon, Jun 27, 2016 at 2:13 PM Bill Deegan <b...@baddogconsulting.com>
wrote:

> Forwarding from distutils mailing list.
> Thought it might be of interest to the community.
>
> -Bill
> Co-Manager SCons Project
> -- Forwarded message ------
> From: Daniel Holth <dho...@gmail.com>
> Date: Sun, Jun 26, 2016 at 1:45 PM
> Subject: [Distutils] enscons, a prototype SCons-powered wheel & sdist
> builder
> To:
>
>
> I've been working on a prototype Python packaging system powered by SCons
> called enscons. https://bitbucket.org/dholth/enscons . It is designed to
> be an easier way to experiment with packaging compared to hacking on
> distutils or setuptools which are notoriously difficult to extend. Now it
> is at the very earliest state where it might be interesting to others who
> are less familiar with the details of pip and Python package formats.
>
> Enscons is able to create wheels and source distributions that can be
> installed by pip. Source distributions built with enscons use
> pyproject.toml from PEP 518 ( https://www.python.org/dev/peps/pep-0518/ )
> to install enscons before setup.py is run, and since pip does not yet
> implement PEP 518, include a setup.py shim for PEP 518. The same
> pyproject.toml includes most of the arguments normally passed to
> setuptools.setup(), which are used to create the metadata files needed by
> pip.
>
> Enscons converts pip's setup.py arguments to SCons ones and then invokes
> SCons, and the project's SConstruct does the rest. For a normal from-pypi
> installation, enscons generates just enough egg_info to let pip build a
> wheel, then pip builds and installs the wheel which has more complete
> metadata.
>
> Of course SCons is more interesting for more complicated projects. I've
> been working on building my pysdl2-cffi binding with SCons. It includes a
> custom Python code generation step that distutils knew nothing about. It's
> practically a one-liner to add my custom build step to SCons, and it knows
> how to rebuild each part of the project only when its dependencies have
> changed. I don't know how to do that with setuptools.
>
> It is still a very early prototype. Don't expect it to be very good. Here
> are some of its limitations:
>
>- There is no error checking.
>- The sdists can only be built inside virtualenvs; otherwise pip
>complains that --user and --target cannot be used together.
>- It also doesn't work if one of the pyproject.toml build requirements
>conflicts with something that's already installed.
>- 'pip install .' doesn't work; it still tries to invoke 'setup.py
>install', which is not implemented.
>- 'setup.py develop' is not implemented. Try PYTHONPATH.
>- I am not an experienced SCons user.
>
> It also relies on what I'm sure will be a short-lived repackaging of SCons
> called 'import_scons' that makes 'python -m SCons' work, and includes a
> Python 3 version of SCons from github.com/timj/scons that seems to work
> well enough for my purposes.
>
> On the plus side, it's short.
>
> If you're interested in implementing Python packaging in SCons, or are
> interested in the straightforward but tedious task of copying just enough
> distutils trickery to implement a robust Python C extension compiler in
> SCons, then perhaps you should check out enscons.
>
> Here is a SConstruct for enscons itself:
>
> import pytoml as tomlimport ensconsmetadata = 
> dict(toml.load(open('pyproject.toml')))['tool']['enscons']
> env = Environment(tools=['default', 'packaging', enscons.generate],   
>PACKAGE_METADATA=metadata,  
> WHEEL_TAG='py2.py3-none-any',  ROOT_IS_PURELIB=True)py_source 
> = Glob('enscons/*.py')
> sdist = env.Package(NAME=env['PACKAGE_NAME'],
> VERSION=env['PACKAGE_METADATA']['version'],PACKAGETYPE='src_zip', 
>target=['dist/' + env['PACKAGE_NAME'] + '-' + env['PACKAGE_VERSION']], 
>source=FindSourceFiles() + ['PKG-INFO', 'setup.py'],)
>
> env.NoClean(sdist)
>
> env.Alias('sdist', sdist)env.Whl('purelib', py_source, root='.')
>
>
>
> Thanks!
>
> ___
> Distutils-SIG maillist  -  distutils-...@python.org
> https://mail.python.org/mailman/listinfo/d

Re: [Scons-dev] posix_spawn

[Daniel Holth]

I don't know why it is slower, just that the profiler says PyPy spends more
time there compared to CPython.

On Mon, Jun 20, 2016 at 12:40 AM Jason Kenny <dragon...@live.com> wrote:

> Why is subprocess to slow?
>
>
>
> We have a monkey patch for this Parts ( I am unsure it was added correctly
> in to SCons yet.. think there was a desire to control the uses of posix
> spawn better) Here is a link:
>
>
> https://bitbucket.org/sconsparts/parts/src/3a389f774f234694994071d784af88c3babaad03/parts/overrides/stubprocess.py?at=master=file-view-default
>
>
>
> We found the posix_spawn was the better default, as cloning the process
> space for a build system ( as well as testing systems) was a speed issue.
>
>
>
>
>
> Jason
>
>
>
> *From:* Scons-dev [mailto:scons-dev-boun...@scons.org] *On Behalf Of *Daniel
> Holth
> *Sent:* Sunday, June 19, 2016 9:55 PM
> *To:* SCons developer list <scons-dev@scons.org>
> *Subject:* [Scons-dev] posix_spawn
>
>
>
> I found this lovely cffi implementation of posix_spawn
> https://github.com/projectcalico/python-posix-spawn . Since
> subprocess.Popen() is too slow on PyPy, this gives a welcome speedup, but
> it would probably help on CPython as well. However the file redirection is
> probably not correct.
>
> in posix.py:
>
> def exec_subprocess_spawnp(l, env):
>
> pid = posix_spawn.posix_spawnp(l[0], l, env)
>
> return os.waitpid(pid, 0)[1]
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

[Scons-dev] posix_spawn

[Daniel Holth]

I found this lovely cffi implementation of posix_spawn
https://github.com/projectcalico/python-posix-spawn . Since
subprocess.Popen() is too slow on PyPy, this gives a welcome speedup, but
it would probably help on CPython as well. However the file redirection is
probably not correct.

in posix.py:

def exec_subprocess_spawnp(l, env):

pid = posix_spawn.posix_spawnp(l[0], l, env)

return os.waitpid(pid, 0)[1]
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

[Scons-dev] success with Tim's Python 3 branch

[Daniel Holth]

Thanks Tim for the Python 3 branch. I was able to run 'bootstrap.py
local-zip' and compile a few CPython extensions without problems.
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] python3-port branch merged to default and then closed

[Daniel Holth]

You have a string regex trying to match bytes.

On Thu, Jun 9, 2016, 14:40 Tim Jenness  wrote:

> On Jun 7, 2016, at 04:46 , Russel Winder  wrote:
>
> On Mon, 2016-06-06 at 15:59 -0700, Tim Jenness wrote:
>
>  […]
>
> Russell — I’m taking a look again at python3 scons. Am I now meant to
> be looking at the default branch for scons on bitbucket? Is there
> work happening on a new branch anywhere?
>
>
> I assume that means me (Russel) :-)
>
>
> Yes. Sorry.
>
> Yes. Bill did a whole lot of work, and so now there is only the
> mainline repository with it's default branch. Any and all changes must
> ensure that all the tests pass when run on Python 2.7.x (I am guessing
> there is a minimum for x here as 2.7 changes so much!), and should
> improve things when run on Python 3.4+.
>
> All pull requests should be against mainline default, and one of the
> core team will either commit it or call for a discussion.
>
>
> I’ve had a quick go at running the bootstrap.py with python3. Here is what
> I’ve learned:
>
> Start by running “futurize -w .” on the entire tree.
>
>
> • When running bootstrap.py it fails when trying to load “rpm” because
> there is a directory called “rpm” in the scons root and that gets imported
> as a namespace package. Renaming “rpm” to “rpm-spec” fixes that.
> • When reading from subprocess you need to decode the bytes (futurize does
> not do that for you). (using .decode() works most of the time).
> • SCons_revision deliberately opens files as binary so the strings being
> inserted have to be encoded to bytes first.
> • similarly write_src_files
> • I’m not entirely sure why there are so many files being opened in binary
> mode that seem to be text files. It would be much less confusing if text
> files were opened in text mode.
> • I can’t build the windows binary targets on my Mac because the mbcs
> encoding is missing so I am skipping that.
> • os.path.walk is missing in python3 and replaced with os.walk (which is
> iterable). os.walk also works on python2.7 and the fix is very easy (and it
> is much cleaner) so should be applied independently of all this other
> python3 work.
>
> With these changes I can get quite a long way through bootstrap and fail
> at :
>
> scons: *** [build/scons-src-2.5.0-stamp] TypeError : can't use a string
> pattern on a bytes-like object
>
> This is after the zipping runs. I don’t know where that error is coming
> from and there is no context other than the attempt to write the stamp
> file. In the bootstrap environment how would I work out how to get a better
> error message? The fix will be easy once I know what line is causing the
> exception.
>
> Weirdly, in another checkout it gets stopped at the Digestify stage with
> md5. (md5 needs bytes: I’m looking at that problem at the moment).
>
> The main issue I see is all the binary file opening going. Almost all the
> problems are related to binary vs str mismatches.
>
> Of course, the futurized scons doesn’t work on python2 as it complains
> early on about an issue with metaclasses. Since I don’t know the scale of
> the python3 issues I looked at those first.
>
> Apologies for this but I’ve been unable to quickly get my head round hg so
> I’ve put scons on github and opened a pull request there. If you want to
> see what I’ve done take a look at: https://github.com/timj/scons/pull/1
> I felt it was better to put the work somewhere rather than have everyone
> rediscover the issues.
>
> As an aside, sometimes kw=[] keyword arguments are used. These are
> dangerous as the same default will be passed into the function each time
> and any modifications to it will persist. Should be kw=None and then assign
> a fresh [] (unless you want it to be persistent of course).
>
> —
> Tim Jenness
>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

[Scons-dev] Using scons as setup.py

[Daniel Holth]

I am working on doing some experiments using scons as the setup.py
implementation. I want to be able to package wheel files more flexibly than
setuptools would allow, and it's nice to have a good, easier to use build
system. A new pip feature will allow dependencies before setup.py is called.

Had to find/replace __slots__ to __slats__ to get it to run in pypy. How
badly do you need slots?

Daniel Holth
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev