Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Glyph Lefkowitz]

> On Feb 12, 2017, at 11:08 AM, Craig Rodrigues  wrote:
> 
> On Tue, Feb 7, 2017 at 11:29 AM, Glyph Lefkowitz  > wrote:
> 
>> On Feb 7, 2017, at 6:59 AM, Jean-Paul Calderone > > wrote:
>> 
>> On Fri, Jan 13, 2017 at 1:13 AM, Glyph Lefkowitz > > wrote:
>> Thanks for highlighting those.  I've put the link in the other direction as 
>> well.
>> 
>> 
>> Craig seems eager to go ahead with reverting this change in behavior.
>> 
>> https://github.com/twisted/twisted/pull/672#issuecomment-275956265 
>> 
>> 
>> As far as I can tell, no one has weighed in on the other side.  So I'm 
>> inclined to go along with the reversion.
> 
> My 2¢ for the other side is: if trial does this, but twist and twistd don't, 
> then it will be possible to get a passing test run for a plugin that doesn't 
> get loaded.  I think it would be simpler and easier to debug to leave these 
> consistent.
> 
> 
> 
> This is an interesting corner case, but I think the twistd and twist issues 
> should be pursued in
> separate discussions and tickets.
> 
> For trial, I would like to proceed with 
> https://github.com/twisted/twisted/pull/672/ 
>  .  With my conversion
> of trial to a console script, the new behavior was unintentional on my part.  
> Since all the unit tests
> passed, I did not notice.  In this ticket: 
> https://twistedmatrix.com/trac/ticket/8978 
> 
> Job-Evers-Meltzer provided a use-case where the new behavior of trial
> broke existing usage.
> 
> trial can be used as a general-purpose tool for running unittest-style tests,
> much like pytest or nose.  The new behavior is disruptive, for minimal 
> benefit,
> so I would like to restore the old behavior.
> 
> I added a unit test for this, so in future, if this breaks again, we will 
> catch it.

There's a lot of controversy around this type of organization of tests; 
personally, I believe it is a horrible antipattern, others, notably Donald 
Stufft, (wrongly) regard it as a best practice.

Apropos of the comment I put on 9035, 
https://twistedmatrix.com/trac/ticket/9035#comment:4 
, would it be acceptable 
for Job Evers‐Meltzer if the syntax were simply 'trial ./tests/' instead?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Craig Rodrigues]

On Tue, Feb 7, 2017 at 11:29 AM, Glyph Lefkowitz 
wrote:

>
> On Feb 7, 2017, at 6:59 AM, Jean-Paul Calderone 
> wrote:
>
> On Fri, Jan 13, 2017 at 1:13 AM, Glyph Lefkowitz 
> wrote:
>
>> Thanks for highlighting those.  I've put the link in the other direction
>> as well.
>>
>>
> Craig seems eager to go ahead with reverting this change in behavior.
>
> https://github.com/twisted/twisted/pull/672#issuecomment-275956265
>
> As far as I can tell, no one has weighed in on the other side.  So I'm
> inclined to go along with the reversion.
>
>
> My 2¢ for the other side is: if trial does this, but twist and twistd
> don't, then it will be possible to get a passing test run for a plugin that
> doesn't get loaded.  I think it would be simpler and easier to debug to
> leave these consistent.
>
>

This is an interesting corner case, but I think the twistd and twist issues
should be pursued in
separate discussions and tickets.

For trial, I would like to proceed with
https://github.com/twisted/twisted/pull/672/ .  With my conversion
of trial to a console script, the new behavior was unintentional on my
part.  Since all the unit tests
passed, I did not notice.  In this ticket:
https://twistedmatrix.com/trac/ticket/8978
Job-Evers-Meltzer provided a use-case where the new behavior of trial
broke existing usage.

trial can be used as a general-purpose tool for running unittest-style
tests,
much like pytest or nose.  The new behavior is disruptive, for minimal
benefit,
so I would like to restore the old behavior.

I added a unit test for this, so in future, if this breaks again, we will
catch it.

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Glyph Lefkowitz]

> On Feb 7, 2017, at 6:59 AM, Jean-Paul Calderone  
> wrote:
> 
> On Fri, Jan 13, 2017 at 1:13 AM, Glyph Lefkowitz  > wrote:
> Thanks for highlighting those.  I've put the link in the other direction as 
> well.
> 
> 
> Craig seems eager to go ahead with reverting this change in behavior.
> 
> https://github.com/twisted/twisted/pull/672#issuecomment-275956265 
> 
> 
> As far as I can tell, no one has weighed in on the other side.  So I'm 
> inclined to go along with the reversion.

My 2¢ for the other side is: if trial does this, but twist and twistd don't, 
then it will be possible to get a passing test run for a plugin that doesn't 
get loaded.  I think it would be simpler and easier to debug to leave these 
consistent.

(I really don't think we should put this behavior back into twist or twistd, 
because "cd malware; twist web --path ." should not pwn your machine.)

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Jean-Paul Calderone]

On Fri, Jan 13, 2017 at 1:13 AM, Glyph Lefkowitz 
wrote:

> Thanks for highlighting those.  I've put the link in the other direction
> as well.
>
>
Craig seems eager to go ahead with reverting this change in behavior.

https://github.com/twisted/twisted/pull/672#issuecomment-275956265

As far as I can tell, no one has weighed in on the other side.  So I'm
inclined to go along with the reversion.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Glyph Lefkowitz]

Thanks for highlighting those.  I've put the link in the other direction as 
well.

> On Jan 12, 2017, at 3:42 PM, Jean-Paul Calderone  
> wrote:
> 
> Note there is still some confusion over this matter.  See 
>  >, 
>  >, and 
>  >.
> 
> Jean-Paul
> 
> On Tue, Jan 3, 2017 at 3:24 PM, Jean-Paul Calderone 
> > wrote:
> On Tue, Jan 3, 2017 at 3:22 PM, Amber "Hawkie" Brown 
> > wrote:
> 
> 
> We have, in the past, fixed up historic NEWS files in later releases (e.g. 
> the one which removed 2.6 support). We could always roll these changes into a 
> post1, but, that seems like a lot of effort. Maybe we could put errata on the 
> blog?
> 
> 
> With a link to the blog post in the NEWS file?  Cold checking blog was below 
> "read git history and try to correlate with the issue tracker" on my list of 
> how to track down this change.
> 
> Jean-Paul
> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Jean-Paul Calderone]

Note there is still some confusion over this matter.  See <
http://twistedmatrix.com/trac/ticket/8972>, <
http://twistedmatrix.com/trac/ticket/8978>, and <
https://github.com/twisted/twisted/pull/672>.

Jean-Paul

On Tue, Jan 3, 2017 at 3:24 PM, Jean-Paul Calderone <
exar...@twistedmatrix.com> wrote:

> On Tue, Jan 3, 2017 at 3:22 PM, Amber "Hawkie" Brown <
> hawk...@atleastfornow.net> wrote:
>
>>
>>
>> We have, in the past, fixed up historic NEWS files in later releases
>> (e.g. the one which removed 2.6 support). We could always roll these
>> changes into a post1, but, that seems like a lot of effort. Maybe we could
>> put errata on the blog?
>>
>>
> With a link to the blog post in the NEWS file?  Cold checking blog was
> below "read git history and try to correlate with the issue tracker" on my
> list of how to track down this change.
>
> Jean-Paul
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Jean-Paul Calderone]

On Tue, Jan 3, 2017 at 3:22 PM, Amber "Hawkie" Brown <
hawk...@atleastfornow.net> wrote:

>
>
> We have, in the past, fixed up historic NEWS files in later releases (e.g.
> the one which removed 2.6 support). We could always roll these changes into
> a post1, but, that seems like a lot of effort. Maybe we could put errata on
> the blog?
>
>
With a link to the blog post in the NEWS file?  Cold checking blog was
below "read git history and try to correlate with the issue tracker" on my
list of how to track down this change.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Amber "Hawkie" Brown]

> On 4 Jan 2017, at 07:20, Glyph Lefkowitz  wrote:
> 
>> 
>> On Jan 2, 2017, at 6:10 AM, Jean-Paul Calderone > > wrote:
>> 
>> On Thu, Sep 1, 2016 at 11:36 AM, Amber "Hawkie" Brown 
>> > wrote:
>> 
>> > On 1 Sep 2016, at 21:42, Yuri > > > wrote:
>> >
>> > Hi all
>> >
>> > I couldn't find Twisted-specific group, so posting here.
>> >
>> > Recently Twisted 16.4.0 got released. Yesterday I've tried to upgrade it 
>> > for my apps and got an error.
>> >
>> > ...
>> >
>> > The errors comes down to this: twistd script does not add current working 
>> > directory to python path (or removes it, I don't know what exactly 
>> > happens), so it fails to import any packages/modules from it. The issue 
>> > does not appear in previous version (Twisted 16.3.2).
>> >
>> > Any ideas what caused it?
>> 
>> Yes -- we moved to using setuptools console scripts, and these console 
>> scripts don't add "." to the PYTHONPATH. We realised this in prerelease but 
>> decided against fixing it, as it adding the current working dir to the PATH 
>> has lead to a lot of  subtle bugs in the past and this is a good chance to 
>> make a break from them.
>> 
>> So, in short, this is expected behaviour -- we generally want people to be 
>> running twistd, trial, etc on *installed* Python packages -- testing or 
>> running from checkouts often hides many bugs about what is or isn't included 
>> in the installed package by accident. If you rely on this behaviour, though, 
>> set the PYTHONPATH environment variable to "." -- e.g. `env PYTHONPATH=. 
>> twistd -n myplugin`.
>> 
>> 
>> FWIW, as a user, it would have been nice to have a NEWS entry for this.  It 
>> would have been easier to discover there than searching through the mail 
>> archives.
> 
> Yes, there definitely should have been.  We only realized the implications of 
> the change after the fact, and only after some discussion decided that it was 
> in fact desirable and should not be rolled back in a patch release.
> 
> We don't have a process for fixing NEWS files right now, and given that 
> releases are immutable, I wonder if there's any way we could.  So, in the 
> rare case that we mess up this way in the future, what (aside from the 
> mailing list) would be a good communication mechanism to users?
> 
> -glyph

We have, in the past, fixed up historic NEWS files in later releases (e.g. the 
one which removed 2.6 support). We could always roll these changes into a 
post1, but, that seems like a lot of effort. Maybe we could put errata on the 
blog?

- Amber___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Glyph Lefkowitz]

> On Jan 2, 2017, at 6:10 AM, Jean-Paul Calderone  
> wrote:
> 
> On Thu, Sep 1, 2016 at 11:36 AM, Amber "Hawkie" Brown 
> > wrote:
> 
> > On 1 Sep 2016, at 21:42, Yuri  > > wrote:
> >
> > Hi all
> >
> > I couldn't find Twisted-specific group, so posting here.
> >
> > Recently Twisted 16.4.0 got released. Yesterday I've tried to upgrade it 
> > for my apps and got an error.
> >
> > ...
> >
> > The errors comes down to this: twistd script does not add current working 
> > directory to python path (or removes it, I don't know what exactly 
> > happens), so it fails to import any packages/modules from it. The issue 
> > does not appear in previous version (Twisted 16.3.2).
> >
> > Any ideas what caused it?
> 
> Yes -- we moved to using setuptools console scripts, and these console 
> scripts don't add "." to the PYTHONPATH. We realised this in prerelease but 
> decided against fixing it, as it adding the current working dir to the PATH 
> has lead to a lot of  subtle bugs in the past and this is a good chance to 
> make a break from them.
> 
> So, in short, this is expected behaviour -- we generally want people to be 
> running twistd, trial, etc on *installed* Python packages -- testing or 
> running from checkouts often hides many bugs about what is or isn't included 
> in the installed package by accident. If you rely on this behaviour, though, 
> set the PYTHONPATH environment variable to "." -- e.g. `env PYTHONPATH=. 
> twistd -n myplugin`.
> 
> 
> FWIW, as a user, it would have been nice to have a NEWS entry for this.  It 
> would have been easier to discover there than searching through the mail 
> archives.

Yes, there definitely should have been.  We only realized the implications of 
the change after the fact, and only after some discussion decided that it was 
in fact desirable and should not be rolled back in a patch release.

We don't have a process for fixing NEWS files right now, and given that 
releases are immutable, I wonder if there's any way we could.  So, in the rare 
case that we mess up this way in the future, what (aside from the mailing list) 
would be a good communication mechanism to users?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] "twistd" in Twisted 16.4.0 can't import modules/packages from current working directory

[Jean-Paul Calderone]

On Thu, Sep 1, 2016 at 11:36 AM, Amber "Hawkie" Brown <
hawk...@atleastfornow.net> wrote:

>
> > On 1 Sep 2016, at 21:42, Yuri  wrote:
> >
> > Hi all
> >
> > I couldn't find Twisted-specific group, so posting here.
> >
> > Recently Twisted 16.4.0 got released. Yesterday I've tried to upgrade it
> for my apps and got an error.
> >
> > ...
> >
> > The errors comes down to this: twistd script does not add current
> working directory to python path (or removes it, I don't know what exactly
> happens), so it fails to import any packages/modules from it. The issue
> does not appear in previous version (Twisted 16.3.2).
> >
> > Any ideas what caused it?
>
> Yes -- we moved to using setuptools console scripts, and these console
> scripts don't add "." to the PYTHONPATH. We realised this in prerelease but
> decided against fixing it, as it adding the current working dir to the PATH
> has lead to a lot of  subtle bugs in the past and this is a good chance to
> make a break from them.
>
> So, in short, this is expected behaviour -- we generally want people to be
> running twistd, trial, etc on *installed* Python packages -- testing or
> running from checkouts often hides many bugs about what is or isn't
> included in the installed package by accident. If you rely on this
> behaviour, though, set the PYTHONPATH environment variable to "." -- e.g.
> `env PYTHONPATH=. twistd -n myplugin`.
>
>
FWIW, as a user, it would have been nice to have a NEWS entry for this.  It
would have been easier to discover there than searching through the mail
archives.

Thanks,
Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python