Re: [Twisted-Python] unified filesystem API (was: SMB server component for twisted)

[Glyph]

> On May 19, 2020, at 5:40 AM, Jean-Paul Calderone  
> wrote:
> 
> Is anyone going to mention twisted.vfs?

You, apparently? :)

If anyone is curious about this code, `git revert 
2059b5313ec236e8c3f77da4bd773ab98299ca1b` to have a look at it.

twisted.vfs was an attempt to do something like this, but it also suffered from 
the split-release infrastructure; it wasn't released so it didn't get user 
feedback, its interfaces were odd in several ways, it was very specific to 
POSIX filesystems, had some ambiguity around whether it wanted to be operating 
in terms of strings or objects, and generally didn't provide much value.

If we want to bring this concept back (as 
https://twistedmatrix.com/trac/ticket/4934 
 alludes to by mentioning 
"twisted.path") it would benefit from lots of discussion and a concerted effort 
to integrate into the existing HTTP/SFTP/FTP code, so we actually create an 
interface that makes it easy for applications to create virtual filesystems 
accessible in multiple ways, with non-filesystem backends, rather than just 
wrapping up our file I/O code in a translation layer between... our versions of 
POSIX constants and the 'os' module's version of those same constants.

Nevertheless 'vfs' may be of archaeological interest for someone embarking on 
such a project; it's not all bad ideas, it just had some problems :).

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API (was: SMB server component for twisted)

[Glyph]

> On May 19, 2020, at 1:54 AM, Barry Scott  wrote:
> 
> I do not see a common class tree works for things that are so different
> as SMB and FTP.

Let's absolutely not try to do this with inheritance :-).  Broadly, I agree 
with John Santos's assessment.  But it's all sort of moot until someone starts 
trying to actually do it and we can talk about a specific proposal!

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Tom Most]

On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:
> On Tue, May 19, 2020 at 8:33 AM Barry Scott  
> wrote:
>> Why not waitpid for all children (pid=0 or pid=-1) in a single call? 
> 
> Because then you might reap a child process someone launched with a different 
> API - eg the Python subprocess module.

Exactly. I am currently using this as a workaround, but it's not a general 
solution. You need to be careful because subprocess use hides in many places. 
For example, beware uuid.uuid1().

---Tom

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API

[John Santos]

On 5/19/2020 4:54 AM, Barry Scott wrote:

On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:

On 16/05/2020 10:55 am, Ian Haywood wrote:

On 15/05/2020 10:52 am, Glyph wrote:

y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega mailto:wsanc...@wsanchez.net>


   I think it's great to get an SMB implementation in the Twisted
org, but why would we even consider adding something like this to
the main Twisted project?

The advantage of twisted itself is cross-protocol abstractions such as
cred. Of course you don't have to be in the repo to use them, but in
practice developers need the discipline of a single project to
maintain consistency, otherwise the human tendency to reinvent wheels
is too strong

Apropos we have two APIs for exporting filesystems which are broadly
similar: SFTP in conch and FTP itself, SMB is about to be a third,
ideally we should have one or have them descend from one another. It's
slightly harder than it sounds as SMB is a file-access, as opposed to
file-transfer, protocol and supports some extra features such as locking.

I do not see a common class tree works for things that are so different
as SMB and FTP.

Barry


I don't think the commonality lies on the network side of things.  I think it is
on the data manipulation side and the local file read/write side, which are 
currently
inherantly synchronous unless you use threads and all that entails.

All three protocols have to open existing files, create new files, read from 
files,
write to files, read file metadata (dates, sizes, etc.), write metadata, 
traverse
directory trees and packetize and send the various types of data and recieve and
depacketize the data.  Only the packetizing and depacketizing items are 
significantly
different between SMB and the other two protocols.  (SMB access to file data 
can be
random or sequential, wherease FTP and SFTP are invariably sequential, but that 
should
easily be accomodated.)



--
John Santos
Evans Griffiths & Hart, Inc.
781-861-0670 ext 539

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Jean-Paul Calderone]

On Tue, May 19, 2020 at 8:33 AM Barry Scott 
wrote:

> On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
>
> >
> > If looking improve process management in Twisted, I think that the
> > lowest-hanging fruit is to watch children with pidfd, rather than reaping
> > all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> > running process --- starts to consume substantial CPU. IIRC it pegged a
> > core at 20,000 processes or so.
>
> Why not waitpid for all children (pid=0 or pid=-1) in a single call?
>
> Barry
>

Because then you might reap a child process someone launched with a
different API - eg the Python subprocess module.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API (was: SMB server component for twisted)

[Jean-Paul Calderone]

Is anyone going to mention twisted.vfs?

On Tue, May 19, 2020 at 8:33 AM Barry Scott 
wrote:

> On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:
> > On 16/05/2020 10:55 am, Ian Haywood wrote:
> > > On 15/05/2020 10:52 am, Glyph wrote:
> > >> y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega  > >> 
> > >>
> > >>>   I think it's great to get an SMB implementation in the Twisted
> > >>> org, but why would we even consider adding something like this to
> > >>> the main Twisted project?
> > >
> > > The advantage of twisted itself is cross-protocol abstractions such as
> > > cred. Of course you don't have to be in the repo to use them, but in
> > > practice developers need the discipline of a single project to
> > > maintain consistency, otherwise the human tendency to reinvent wheels
> > > is too strong
> >
> > Apropos we have two APIs for exporting filesystems which are broadly
> > similar: SFTP in conch and FTP itself, SMB is about to be a third,
> > ideally we should have one or have them descend from one another. It's
> > slightly harder than it sounds as SMB is a file-access, as opposed to
> > file-transfer, protocol and supports some extra features such as locking.
>
> I do not see a common class tree works for things that are so different
> as SMB and FTP.
>
> Barry
>
>
>
> >
> > Ian
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Barry Scott]

On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
> On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
> >> In the one case that may be complicated I think that pinning Twisted will
> >> be acceptable, since the service in question is a process supervisor
> >> without direct external exposure.> 
> > Well in that case I'll just do my best to add lots of goodies (posix_spawn
> > support!) that will make you want to upgrade that to current twisted and
> > python 3.8 :).
> Oh it's not for lack of wanting. :-)
> 
> The service in question is actually spawning Python processes, so
> posix_spawn support doesn't seem likely to help much: fork() + exec() is a
> drop in the bucket compared to Python imports, particularly in Python 3
> (I'm hoping to explore oxidized-importer [1] to help with this at some
> point).

Would putting all the .pyc/.pyo into a zip file help with the import problem?
You only have one file to open, the .zip, and no compile as its only 
.pyc/.pyo.


> 
> If looking improve process management in Twisted, I think that the
> lowest-hanging fruit is to watch children with pidfd, rather than reaping
> all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> running process --- starts to consume substantial CPU. IIRC it pegged a
> core at 20,000 processes or so.

Why not waitpid for all children (pid=0 or pid=-1) in a single call?

Barry
> 
> ---Tom
> 
> [1]: https://pypi.org/project/oxidized-importer/




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] unified filesystem API (was: SMB server component for twisted)

[Barry Scott]

On Saturday, 16 May 2020 02:14:57 BST Ian Haywood wrote:
> On 16/05/2020 10:55 am, Ian Haywood wrote:
> > On 15/05/2020 10:52 am, Glyph wrote:
> >> y 14, 2020, at 5:23 PM, Wilfredo Sánchez Vega  >> 
> >> 
> >>>   I think it's great to get an SMB implementation in the Twisted
> >>> org, but why would we even consider adding something like this to
> >>> the main Twisted project?
> > 
> > The advantage of twisted itself is cross-protocol abstractions such as
> > cred. Of course you don't have to be in the repo to use them, but in
> > practice developers need the discipline of a single project to
> > maintain consistency, otherwise the human tendency to reinvent wheels
> > is too strong
> 
> Apropos we have two APIs for exporting filesystems which are broadly
> similar: SFTP in conch and FTP itself, SMB is about to be a third,
> ideally we should have one or have them descend from one another. It's
> slightly harder than it sounds as SMB is a file-access, as opposed to
> file-transfer, protocol and supports some extra features such as locking.

I do not see a common class tree works for things that are so different
as SMB and FTP.

Barry



> 
> Ian
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Richard van der Hoff]

On 16/05/2020 06:56, Glyph wrote:



On May 15, 2020, at 8:40 PM, Craig Rodrigues > wrote:


Maybe it would be OK to do one more release of Twisted and announce 
that as the last release supporting Python 3.5, before

dropping support?


Yeah; whenever we drop a Python version we should always support at 
least one more release, so that people have some notice before they 
lose access to the next set of security updates.


Any 3.5 users on this list who would want to postpone it longer than this?

Sadly we have an important customer whose servers run debian oldstable, 
which means we need to stay compatible with 3.5 until we can persuade 
them to upgrade, and it's taken a couple of years to get them off python 
2.7...


I'm not sure that should necessarily affect your plans, but I doubt 
we're alone in this situation.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python