Re: [Scons-dev] Subprocess.popen() on win32?

2018-10-07 Thread Jason Kenny 
On Linux I did not use the escaping function It seemed to make it worse. Popen 
has some code that does escaping.

>> Do we lose redirects with swapping in the Popen() ?

I was using Popen as I could redirect. The current code might redirect, but 
there is no control over how it works that I understand…

If you look at:
https://bitbucket.org/sconsparts/parts/src/master/parts/part_logger.py

you will see what is  do. There is more code to support this and honestly there 
is some clean up to do. But it works correctly. There is basically a pipe 
redirector class that exists and is passed on to the stdout stderr pipes of 
Popen() each of these are running in a thread and do nothing but grab output 
that is then given to a reporter class ( this is the area of have some clean up 
to do) that allows all output to be written to the console and to go through 
the logger classes I have to output the text to a given log file ( in my case I 
have per component loggers ( ie for the commands that happen in a given 
component) and a global logger that has everything as it comes. The only effect 
this really has on the output is that -j builds will print whole lines of text, 
vs mixed up lines of text from different lines.

Oh I also see I have an updated killtree functions to shut down better and a 
timeout value ( so a command does not lock up the build, which is great for 
running tests). I have an updated version of the kill logic I have to add to 
this in a different project of mine I need to sync with this one.


Jason

From: Scons-dev  On Behalf Of Bill Deegan
Sent: Sunday, October 7, 2018 2:47 PM
To: SCons developer list 
Subject: Re: [Scons-dev] Subprocess.popen() on win32?


On Sun, Oct 7, 2018 at 2:25 PM Jason Kenny 
mailto:dragon...@live.com>> wrote:

I can see what I can do here. I have to make a new Drop of Parts first ( having 
a regression in my code I have to fix )

I don’t believe there should be any limitation. The only issue I believe I had 
was with moving to python 27 as there was a change from 26 in quote handling. 
However we are on 2.7 or better so this should not be an issue. Oh there was 
one other thing. I have to ignore the “escape” function that is passed in the 
SPAWN callback. It breaks with 2.7 on Window and Linux. I also removed the 
Scons override for the open call as it was messing up writing to the console 
device ( Plus I needed to override it to allow symbolic and hard links to work 
on win32)

If we're using subprocess.popen() then is escaping needed on linux?

Which lead to a question for me on do we have an official way in SCons to get 
the stdout and stderr text from an Action that was called. I don’t recall 
having this before and as such I used the SPAWN to help me add logging and 
colorized text in Parts.

As far as I know there's currently no supported way to capture stderr or stdout 
from an action. (you can redirect in your command to a file).

Do we lose redirects with swapping in the Popen() ?

-Bill


Jason


From: Scons-dev 
mailto:scons-dev-boun...@scons.org>> On Behalf Of 
Bill Deegan
Sent: Friday, October 5, 2018 12:44 PM
To: SCons developer list mailto:scons-dev@scons.org>>
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

Jason,

Any limitations with doing this over the current code?
Could you craft a pull request for this?

-Bill

On Fri, Oct 5, 2018 at 10:37 AM Jason Kenny 
mailto:dragon...@live.com>> wrote:
Been using popen with parts via overriding SPAWN var since the start.
+1 on having this be the default
Jason
Get Outlook for 
Android


From: Scons-dev 
mailto:scons-dev-boun...@scons.org>> on behalf of 
Bill Deegan mailto:b...@baddogconsulting.com>>
Sent: Friday, October 5, 2018 10:32:50 AM
To: SCons developer list
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

Come now.
Win32.py's spawn implementation was updated in 2012.. ;)

If we universally switch to subprocess.Popen(), then we can safely chdir for 
non python function actions.
But passing in the cwd argument.

That seems likely to be useful..

-Bill


On Fri, Oct 5, 2018 at 10:28 AM Russel Winder 
mailto:rus...@winder.org.uk>> wrote:
I shall be flippant in the hope of being humorous, yet with an element of
possible truth:

Perhaps no-one has bothered to update that bit of code in the last decade and
a half?


Did you mean subprocess.Popen? Worth noting that the API for subprocess
changes from call* in earlier versions to run in later. Somewhat annoying for
compatibility across many Python versions.


On Fri, 2018-10-05 at 10:17 -0700, Bill Deegan wrote:
> Greetings,
>
> Any reason we're not using subprocess.popen() on win32?
> We're still using os.spawnve() which is known to have some threading issues.
>

Re: [Scons-dev] Subprocess.popen() on win32?

2018-10-07 Thread Bill Deegan 
On Sun, Oct 7, 2018 at 2:25 PM Jason Kenny  wrote:

>
>
> I can see what I can do here. I have to make a new Drop of Parts first (
> having a regression in my code I have to fix )
>
>
>
> I don’t believe there should be any limitation. The only issue I believe I
> had was with moving to python 27 as there was a change from 26 in quote
> handling. However we are on 2.7 or better so this should not be an issue.
> Oh there was one other thing. I have to ignore the “escape” function that
> is passed in the SPAWN callback. It breaks with 2.7 on Window and Linux. I
> also removed the Scons override for the open call as it was messing up
> writing to the console device ( Plus I needed to override it to allow
> symbolic and hard links to work on win32)
>

If we're using subprocess.popen() then is escaping needed on linux?

>
>
> Which lead to a question for me on do we have an official way in SCons to
> get the stdout and stderr text from an Action that was called. I don’t
> recall having this before and as such I used the SPAWN to help me add
> logging and colorized text in Parts.
>

As far as I know there's currently no supported way to capture stderr or
stdout from an action. (you can redirect in your command to a file).

Do we lose redirects with swapping in the Popen() ?

-Bill


>
>
> Jason
>
>
>
>
>
> *From:* Scons-dev  *On Behalf Of *Bill Deegan
> *Sent:* Friday, October 5, 2018 12:44 PM
> *To:* SCons developer list 
> *Subject:* Re: [Scons-dev] Subprocess.popen() on win32?
>
>
>
> Jason,
>
>
>
> Any limitations with doing this over the current code?
>
> Could you craft a pull request for this?
>
>
>
> -Bill
>
>
>
> On Fri, Oct 5, 2018 at 10:37 AM Jason Kenny  wrote:
>
> Been using popen with parts via overriding SPAWN var since the start.
>
> +1 on having this be the default
>
> Jason
>
> Get Outlook for Android
> 
>
>
> --
>
> *From:* Scons-dev  on behalf of Bill Deegan <
> b...@baddogconsulting.com>
> *Sent:* Friday, October 5, 2018 10:32:50 AM
> *To:* SCons developer list
> *Subject:* Re: [Scons-dev] Subprocess.popen() on win32?
>
>
>
> Come now.
>
> Win32.py's spawn implementation was updated in 2012.. ;)
>
>
>
> If we universally switch to subprocess.Popen(), then we can safely chdir
> for non python function actions.
>
> But passing in the cwd argument.
>
>
>
> That seems likely to be useful..
>
>
>
> -Bill
>
>
>
>
>
> On Fri, Oct 5, 2018 at 10:28 AM Russel Winder 
> wrote:
>
> I shall be flippant in the hope of being humorous, yet with an element of
> possible truth:
>
> Perhaps no-one has bothered to update that bit of code in the last decade
> and
> a half?
>
>
> Did you mean subprocess.Popen? Worth noting that the API for subprocess
> changes from call* in earlier versions to run in later. Somewhat annoying
> for
> compatibility across many Python versions.
>
>
> On Fri, 2018-10-05 at 10:17 -0700, Bill Deegan wrote:
> > Greetings,
> >
> > Any reason we're not using subprocess.popen() on win32?
> > We're still using os.spawnve() which is known to have some threading
> issues.
> > https://bugs.python.org/issue6476
> 
> >
> > -Bill
> > ___
> > Scons-dev mailing list
> > Scons-dev@scons.org
> > https://pairlist2.pair.net/mailman/listinfo/scons-dev
> 
> --
> Russel.
> ===
> Dr Russel Winder  t: +44 20 7585 2200
> 41 Buckmaster Roadm: +44 7770 465 077
> London SW11 1EN, UK   w: www.russel.org.uk
> 
>
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
> 
>
>

Re: [Scons-dev] should src/engine/SCons/Tool/docbook/util/* be included in the packages?

2018-10-07 Thread Bill Deegan 
Are the docbook xslt stylesheets we're currently including reasonably
available from distro's repos?


On Sun, Oct 7, 2018 at 2:08 PM Dirk Bächle  wrote:

> Hi there,
>
> and sorry for the late answer...
>
> On 06.10.2018 06:26, Bill Deegan wrote:
> > Strange that it's referring to
> "/usr/lib/scons/SCons/Tool/docbook/utils/xmldepend.xsl",  it should be
> referring to build/... for the
> > unpacked version of the package.
> >
> > Have you run python bootstrap.py first?
> >
> > On Fri, Oct 5, 2018 at 1:35 PM Paweł Tomulik  > wrote:
> >
> > The problem still persists:
> >
> > I/O warning : failed to load external entity
> > "/usr/lib/scons/SCons/Tool/docbook/utils/xmldepend.xsl"
> > scons: *** [build/doc/user/manual.fo ]
> parserError : xmlParseFile() failed
> > scons: building terminated because of errors.
> >
> > should I file an issue on GitHub?
> >
>
> I think that xmldepend.xsl should be contained in the packages. I checked
> back to SCons v2.3.2 and it was never included right from
> the start. Funny that nobody noticed until now, I guess I always tried to
> rebuild from the sources on my machine...definitely have
> to amend my workflow for preparing release packages. ;)
>
> Pawel, please open an issue on github about this, I'll try to look into it
> as soon as possible.
>
> We should also discuss what to do about the copy of the docbook XSLT
> stylesheets that are checked in alongside with the sources.
> Should these be packaged too, such that the user doesn't have to care
> about installing them separately?
>
> Best regards,
>
> Dirk
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] Subprocess.popen() on win32?

2018-10-07 Thread Jason Kenny 

I can see what I can do here. I have to make a new Drop of Parts first ( having 
a regression in my code I have to fix )

I don’t believe there should be any limitation. The only issue I believe I had 
was with moving to python 27 as there was a change from 26 in quote handling. 
However we are on 2.7 or better so this should not be an issue. Oh there was 
one other thing. I have to ignore the “escape” function that is passed in the 
SPAWN callback. It breaks with 2.7 on Window and Linux. I also removed the 
Scons override for the open call as it was messing up writing to the console 
device ( Plus I needed to override it to allow symbolic and hard links to work 
on win32)

Which lead to a question for me on do we have an official way in SCons to get 
the stdout and stderr text from an Action that was called. I don’t recall 
having this before and as such I used the SPAWN to help me add logging and 
colorized text in Parts.

Jason


From: Scons-dev  On Behalf Of Bill Deegan
Sent: Friday, October 5, 2018 12:44 PM
To: SCons developer list 
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

Jason,

Any limitations with doing this over the current code?
Could you craft a pull request for this?

-Bill

On Fri, Oct 5, 2018 at 10:37 AM Jason Kenny 
mailto:dragon...@live.com>> wrote:
Been using popen with parts via overriding SPAWN var since the start.
+1 on having this be the default
Jason
Get Outlook for 
Android


From: Scons-dev 
mailto:scons-dev-boun...@scons.org>> on behalf of 
Bill Deegan mailto:b...@baddogconsulting.com>>
Sent: Friday, October 5, 2018 10:32:50 AM
To: SCons developer list
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

Come now.
Win32.py's spawn implementation was updated in 2012.. ;)

If we universally switch to subprocess.Popen(), then we can safely chdir for 
non python function actions.
But passing in the cwd argument.

That seems likely to be useful..

-Bill


On Fri, Oct 5, 2018 at 10:28 AM Russel Winder 
mailto:rus...@winder.org.uk>> wrote:
I shall be flippant in the hope of being humorous, yet with an element of
possible truth:

Perhaps no-one has bothered to update that bit of code in the last decade and
a half?


Did you mean subprocess.Popen? Worth noting that the API for subprocess
changes from call* in earlier versions to run in later. Somewhat annoying for
compatibility across many Python versions.


On Fri, 2018-10-05 at 10:17 -0700, Bill Deegan wrote:
> Greetings,
>
> Any reason we're not using subprocess.popen() on win32?
> We're still using os.spawnve() which is known to have some threading issues.
> https://bugs.python.org/issue6476
>
> -Bill
> ___
> Scons-dev mailing list
> Scons-dev@scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
--
Russel.
===
Dr Russel Winder  t: +44 20 7585 2200
41 Buckmaster Roadm: +44 7770 465 077
London SW11 1EN, UK   w: 
www.russel.org.uk

___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

Re: [Scons-dev] should src/engine/SCons/Tool/docbook/util/* be included in the packages?

2018-10-07 Thread Dirk Bächle 

Hi there,

and sorry for the late answer...

On 06.10.2018 06:26, Bill Deegan wrote:
Strange that it's referring to "/usr/lib/scons/SCons/Tool/docbook/utils/xmldepend.xsl",  it should be referring to build/... for the 
unpacked version of the package.


Have you run python bootstrap.py first?

On Fri, Oct 5, 2018 at 1:35 PM Paweł Tomulik mailto:ptomu...@meil.pw.edu.pl>> wrote:

The problem still persists:

I/O warning : failed to load external entity
"/usr/lib/scons/SCons/Tool/docbook/utils/xmldepend.xsl"
scons: *** [build/doc/user/manual.fo ] parserError : 
xmlParseFile() failed
scons: building terminated because of errors.

should I file an issue on GitHub?



I think that xmldepend.xsl should be contained in the packages. I checked back to SCons v2.3.2 and it was never included right from 
the start. Funny that nobody noticed until now, I guess I always tried to rebuild from the sources on my machine...definitely have 
to amend my workflow for preparing release packages. ;)


Pawel, please open an issue on github about this, I'll try to look into it as 
soon as possible.

We should also discuss what to do about the copy of the docbook XSLT stylesheets that are checked in alongside with the sources. 
Should these be packaged too, such that the user doesn't have to care about installing them separately?


Best regards,

Dirk
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev

[Scons-dev] [Fwd: Re: Help needed for D support in SCons on Windows]

2018-10-07 Thread Russel Winder 
 Forwarded Message 
From: rjframe via Digitalmars-d 
Reply-To: digitalmars.D 
To: digitalmar...@puremagic.com
Subject: Re: Help needed for D support in SCons on Windows
Date: Sun, 7 Oct 2018 10:51:44 + (UTC)
Newsgroups: digitalmars.D

> On Tue, 02 Oct 2018 06:51:10 +0100, Russel Winder wrote:
> 
> > It turns out there are a number problems with the SCons tests running on
> > Windows, many of them associated with the D support.
> > 
> > https://github.com/SCons/scons/issues/3205
> > 
> > As you will see I am choosing not to get involved in anything to do with
> > Windows. The D support in SCons on Windows does though need some work,
> > and so SCons needs some people to get involved – due to enlightened
> > self-interest.
> 
> I don't use SCons, but took a quick look; my current guess is that the 
> problem is the chocolatey installation of dmd and the rest in AppVeyor; 
> chocolatey doesn't install DMD in %SYSTEMROOT% and I don't know if the 
> path updates take effect without restarting the shell (my own PC is too 
> polluted to test this locally).
> 
> Using SCons.Debug.Trace just crashes with "ValueError: I/O operation on 
> closed file.", writing both to stdout and a file, so I haven't been able 
> to test anything.
> 
> --Ryan
-- 
Russel.
===
Dr Russel Winder  t: +44 20 7585 2200
41 Buckmaster Roadm: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk



signature.asc
Description: This is a digitally signed message part
___
Scons-dev mailing list
Scons-dev@scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev