Re: [Twisted-Python] Release blocker: Use latest pydoctor release ?

[Maarten ter Huurne]

On Sunday, 28 February 2021 23:04:49 CET Craig Rodrigues wrote:

> With respect to API docs, I am not as familiar with the whole process,
> with how they are generated
> and what are doing with readthedocs vs. docs on twistedmatrix.com.
> API docs are generated and don't live in the source tree.
> 
> For example:
> https://twistedmatrix.com/documents/current/api/twisted.html
> 
> Is the long term direction to get rid of that, and point everything at
> readthedocs?

Yes, as far as I know the intention is to reduce the amount of 
infrastructure that has to be maintained by Twisted developers.

Adi is doing the actual work for the migration; I only contribute 
indirectly by reviewing PRs that make pydoctor integrate better with 
Sphinx.

> Since you have done a lot of work in this area, can you shed some
> light on what you think the future direction of all this stuff should
> be with respect to the API docs?

There are efforts to make the output of pydoctor more user friendly. 
This is mainly done by Tristan, but I occasionally work on it as well. 
In the next major release we should have a clearer presentation of 
parameter types, more navigation links and, if it's ready in time, a 
side bar containing a page outline.


The main thing I'm working on is improving accuracy. Currently pydoctor 
can draw conclusions based on incomplete information if import cycles 
are present and with type annotations being added, there are more import 
cycles than before. To solve this, I want to separate the parsing and 
local analysis from the analysis that involves multiple documented 
objects, and run the latter only after all of the former is finished.

On Twisted's side of improving accuracy, we currently have a lot of 
'@type' fields in the docstrings which aren't correct. In particular, 
there is still a lot of fallout from the Python 2 to 3 migration, where 
a documented type of 'str' can either mean 'bytes' or 'str'. I think 
that most '@type' fields should be replaced by type annotations, in 
which case mypy will verify that the documented type matches the type 
that the code is actually using.

Something that might help here is to generate type stubs from the 
information found in docstrings and then automatically apply those stubs 
to Twisted's code. A possible implementation would be for pydoctor to 
export the result of docstring parsing as JSON and then a separate tool 
could generate type stubs from that data.


Twisted's customizations to pydoctor are a bit of a pain, since changes 
in pydoctor break Twisted's API docs quite often. The customizations 
exist in two parts: code and templates.

For the code customizations, these are done using subclassing, so the 
tight coupling makes it hard to change pydoctor's design without 
breaking the customizations. If possible, I'd prefer to remove the need 
for these customizations. If that isn't possible, we'd have to design a 
new plugin interface that is a lot more shielded from pydoctor's 
internals.

For anyone interested, the details are discussed here:
https://github.com/twisted/pydoctor/issues/315

For the template customizations, we're splitting up the templates into 
smaller chunks. This will eliminate or at least greatly reduce the 
amount of copy-pasted template content, which should allow Twisted to 
switch to new major pydoctor reeleases without having to sync the 
template content changes almost every time.

The PR for the template rework is here:
https://github.com/twisted/pydoctor/issues/299

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Release blocker: Use latest pydoctor release ?

[Maarten ter Huurne]

On Sunday, 28 February 2021 22:02:48 CET Craig Rodrigues wrote:
> That sounds like a good plan.  Thanks for working on this.
> 
> Keep in mind the points regarding docs raised by Glyph here:
> 
> 
> https://twistedmatrix.com/pipermail/twisted-python/2021-February/06543
> 2.html

Those are good points, but I meant API docs specifically and I don't 
think there should be any overlap between the wiki and the API docs. If 
I'm mistaken, please let me know.

Bye,
Maarten

> --
> Craig
> 
> 
> On Sunday, February 28, 2021, Maarten ter Huurne
> 
> wrote:
> > However, I think that it would be preferable to get new docs out as
> > soon as possible, rather than waiting for even better docs. Once
> > those improvements are integrated, we could do another pydoctor
> > release and update Twisted's API docs without there being a new
> > Twisted release.
> > 
> > Does this sound like a plan?



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Release blocker: Use latest pydoctor release ?

[Maarten ter Huurne]

On Sunday, 28 February 2021 19:26:16 CET I wrote:

> I'll try to get a pydoctor release out today from the current master
> branch. While the release publishing is automated, writing the release
> notes is not, but that shouldn't take too long.

pydoctor 21.2.0 is now available on PyPI:
https://pypi.org/project/pydoctor

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Release blocker: Use latest pydoctor release ?

[Maarten ter Huurne]

On Sunday, 28 February 2021 08:20:22 CET Craig Rodrigues wrote:
> Thanks for the clarification.
> 
> You have provided a lot of useful details.  There are a lot of
> interconnected pieces, with
> multiple people working on different things.
> 
> For now, I am going to proceed to finish the release as-is.
> 
> The documentation may not be automatically deployed as part of the
> release, but I think that can be done as a follow-on task.

I'll try to get a pydoctor release out today from the current master 
branch. While the release publishing is automated, writing the release 
notes is not, but that shouldn't take too long.

There are some improvements in the works that aren't ready to merge yet 
but would improve Twisted's documentation, both in presentation (better 
navigation and presentation of types) and in accuracy (better handling 
of cyclic imports, which are becoming more common with the adoption of 
type annotations).

However, I think that it would be preferable to get new docs out as soon 
as possible, rather than waiting for even better docs. Once those 
improvements are integrated, we could do another pydoctor release and 
update Twisted's API docs without there being a new Twisted release.

Does this sound like a plan?

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] pydoctor 20.7.0

[Maarten ter Huurne]

Hi all,

I'm happy to announce that a new release of pydoctor, the documentation 
extractor used by Twisted, is available on PyPI.

https://pypi.org/project/pydoctor/

Thanks to everyone who contributed with patches and reviews!


Major changes in this release:

- Python 3 support
- Type annotations on attributes are supported when running on Python 3
- Type comments on attributes are supported when running on Python 3.8+
- Type annotations on function definitions are not supported yet
- Undocumented attributes are now included in the output
- Attribute docstrings: a module, class or instance variable can be 
documented by a following it up with a docstring
- Improved error reporting: more errors are reported, error messages 
include file name and line number
- Dropped support for implicit relative imports
- Explicit relative imports (using "from") no longer cause warnings
- Dropped support for index terms in epytext ("X{}"); this was never 
supported in any meaningful capacity, but now the tag is gone

This will be the last release to support Python 2.7 and 3.5: future 
releases will require Python 3.6 or later.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] mypy integrated with CI for twisted

[Maarten ter Huurne]

On Thursday, 25 June 2020 14:18:04 CEST Adi Roiban wrote:
> On Wed, 24 Jun 2020 at 13:48, Jean-Paul Calderone
> 
> wrote:
> > On Wed, Jun 24, 2020 at 12:44 AM Glyph  
wrote:
> >> On Jun 23, 2020, at 5:34 AM, Adi Roiban  wrote:
> >> 
> >> Hi Craig,
> >> 
> >> On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues
> >> >> 
> >> wrote:
> >>> I have merged some more fixes for mypy to Twisted trunk branch.
> >>> 
> >>> In trunk, you can run mypy with:
> >>> 
> >>> *tox -e mypy*
> >>> 
> >>> Currently this results in *171* errors, which is way down from
> >>> >1000
> >>> errors
> >>> a month ago.
> >>> 
> >>> In addition, if you look at any new PR's there is a *Mypy Ubuntu*
> >>> job
> >>> running on Azure pipeline, which runs mypy.  Right now errors from
> >>> this job
> >>> are ignored and does not block the PR.  However, if we can get the
> >>> mypy errors down to zero, we can make mypy status a blocker for
> >>> the PR.>> 
> >> Thanks for working on this.
> >> 
> >> Looking forward to have a real green mypy build.
> >> 
> >> A general question: Why Twisted used Azure Devops and not GitHub
> >> actions?
> >> 
> >> 
> >> Azure Pipelines gave us substantially more parallel capacity than
> >> is
> >> available via Github Actions, which means we can make build
> >> statuses appear much sooner.  Plus they support more platforms.
> > 
> > Does Twisted have a special deal with Azure Pipelines?  Or is the
> > use of past-tense in this sentence intentional? :)  Or are the docs
> > for the respective platforms wrong/misleading?
> > 
> > 
> > https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/co
> > ncurrent-jobs?view=azure-devops says free-tier public projects get
> > 10 parallel jobs.
> > 
> > 
> > https://help.github.com/en/actions/getting-started-with-github-actio
> > ns/about-github-actions#usage-limits says free tier projects get 20
> > parallel jobs.
> > 
> > (Of course this says nothing about the number of supported
> > platforms.)
> My understanding is that GitHub actions are free for public
> repositories.
> 
> My suggestion is to use Azure Pipelines and Travis for the main trial
> tests and use Circle-CI or GitHub Actions for the other tests.
> GitHub Actions has a nice integration with a GitHub PR and you can
> check the results without having to navigate to a different page.
> And with GitHub actions you can add any new workflow without extra
> permission to  Azure Devop.
> 
> With GitHub actions for free and available on LInux/Windows/macOS , I
> am not sure if keeping Circle-Ci makes sense.

One problem with the Circle CI runs for Twisted that I ran into recently 
is that it won't let me view the results of runs unless I grant it a 
bunch of permissions it doesn't need, including read/write access to all 
my repositories.

Since that is an unnecessary security risk, I refused, but that does 
mean that I can't view some of the CI results, which is a pain when one 
of those checks fails.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Running tests with monkeytype

[Maarten ter Huurne]

On Saturday, 20 June 2020 06:51:21 CEST Moshe Zadka wrote:
> Hi all,
> 
> If you want to get a first rough draft of types for mypy, has anyone
> tried running the tests under monkeytype[1]?
> 
> Moshe Z.
> 
> [1] https://monkeytype.readthedocs.io/en/stable/

That would be worth running in any case, if only as a reference.

Whether it would be worth applying as-is, I'm not sure: monkeytype would 
capture the concrete type passed to functions, while in the type 
annotations we'd want to have Zope interfaces and abstract types 
(Iterable etc) instead.

With some scripted filtering, maybe we can apply part of monkeytype's 
captured types automatically. We don't need to automate 100%; any 
substantial amount will save a lot of manual work.


Another thing we could do to automate annotation is extract the 
documented types from the docstrings. For example by modifying pydoctor.

Which also brings up the question: in the long term (*), do we want to 
have both type annotations and types in the docstring? Or do we only 
want to document types in docstrings if there is no exact annotations 
possible in Python's type system?

(*) Currently, pydoctor uses Python 3.6 style ("var: type") annotations 
for variables, but it ignores annotations for functions and it ignores 
type comments. Function annotations are on my to-do list, while type 
comments are supported by Python 3.8's ast module. So eventually, all 
annotations should end up in the output of pydoctor.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Question about FileDescriptor.loseConnection() signature found by mypy

[Maarten ter Huurne]

On Tuesday, 16 June 2020 10:51:21 CEST Glyph wrote:
> > On Jun 15, 2020, at 8:43 PM, Craig Rodrigues
> >  wrote:
> > 
> > In twisted.internet.abstract.FileDescriptor.loseConnection, the
> > loseConnection method> 
> > is defined like:
> > def loseConnection(self, 
_connDone=failure.Failure(main.CONNECTION_DONE)):
> I think this signature might just be... wrong?  ITransport doesn't
> include it.  Does anything actually use this argument?

I think this isn't actually an argument: the underscore in the name 
suggests it is not part of the interface. What seems to be happening 
here is that twisted.internet.abstract.FileDescriptor.loseConnection 
defines a constant by putting it in an argument default value.

A simple solution would be to replace it by a class-scope or module-
scope constant.


By the way, is it valid to wrap an exception that was never raised in a 
Failure object? Because that is what happens here by reusing the single 
_connDone instance. It seems Failure's implementation is prepared to 
handle it, but its docstring doesn't mention it.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Inheriting DatagramProtocol, which is an old-style class

[Maarten ter Huurne]

On Saturday, 8 February 2020 01:50:04 CET Go Luhng wrote:
> I am creating a child class
> `Child(twisted.internet.protocol.DatagramProtocol)` to implement a
> custom UDP multicast protocol.
> 
> I need to add an `__init__()` to `Child`, but as part of that I need
> to call `super()` which is impossible because `DatagramProtocol` is
> old-style.
> 
> How should I proceed?

There is no __init__() in DatagramProtocol or in its superclass 
AbstractDatagramProtocol, so you can just skip the call.

> Also, why are there old-style classes in the latest release of
> Twisted?

I don't know the reason, but it seems to be deliberate, since 
AbstractDatagramProtocol is annotated with @_oldStyle in the source.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] sharing a dict between child processes

[Maarten ter Huurne]

On Wednesday, 6 November 2019 07:19:56 CET Waqar Khan wrote:
> Hi,
> So, I am writing a twisted server. This server spawn multiple child
> processes using reactor spawnProcess that initializes a process
> protocol.
> 
> Now, each of the childprocess receives some REST requests. Each
> process has a dict that acts as cache.
> Now, I want to share dict across processes.
> In general, python has SharedMemoryManager in multiprocessing module
> which would have helped.
> https://docs.python.org/3/library/multiprocessing.shared_memory.html#m
> ultiprocessing.managers.SharedMemoryManager.SharedMemory But since I
> am using twisted internal process implementation, how do I share this
> dict across the processes so that all the processes use this common
> cache?

Keeping a dictionary in SharedMemoryManager seems far from trivial. I 
don't think you can allocate arbitrary Python objects in the shared 
memory and even if you could, you would run into problems when one 
process mutates the dictionary while another is looking up something or 
also mutating it.

It could in theory work if you implement a custom lock-less dictionary, 
but that would be a lot of work and hard to get right. Also having 
shared memory mutations be synced between multiple CPU cores could 
degrade performance, since keeping core-local CPU caches in sync is 
expensive.

Would it be an option to have only one process accept the REST requests, 
check whether the result is in the cache and only distribute work to the 
other processes if you get a cache miss? Typically the case where an 
answer is cached is pretty fast, so perhaps you don't need multiple 
processes to handle incoming requests.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] In memory cache in twisted

[Maarten ter Huurne]

On Friday, 27 September 2019 05:48:35 CEST Waqar Khan wrote:
> Hi Maarten,
>I think you have hit the problem in the head. I do think this is
> feasible as I have observed that as size of cache increases, things do
> get better which might support your theory.
> 
> Is there a simple example you can add on "put a Deferred for the fetch
> operation ". I am really just getting started with twisted.
> Thanks for all the help.

Unfortunately I don't think I have any code lying around that implements 
this kind of cache.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] In memory cache in twisted

[Maarten ter Huurne]

On Friday, 27 September 2019 04:38:46 CEST Waqar Khan wrote:
> Hi,
>   What's a good way to use a simple dictionary as a cache in twisted
> framework?
> Basically, I have this callback chain where I ultimately make a rest
> call (in non-blocking way using treq) to fetch some data. But before
> I make the call, I am using a dictionary to see if the value is
> available or not. But, I have noticed that the event loop gets pretty
> busy(sometimes, things get stuck and twisted server stops) as soon as
> I add this logic.. Which is pretty much
> 
> @defer.inlinecallbacks
> def fetch(key):
>   if key in cache:
>return cache[key]
>   # else call back to treq to fetch value
>cache[key] = value
>return value
> 
> This dict can grow to around 50k.. What's a good way to solve this
> issue?

If it gets stuck, then the cause for that is probably in the part of the 
code you omitted. So it would help to elaborate on how the value is 
fetched exactly.

I can see two other problems with this caching mechanism though:

1. Items are never removed from the cache, so unless there is a limit to 
the number of different keys that can be used, the cache can grow 
indefinitely. You might want something like an LRU cache rather than a 
plain dictionary.

https://docs.python.org/3/library/functools.html#functools.lru_cache

2. If a lot of clients are requesting the same thing, you won't see any 
benefits from caching until the first request completes. So you could 
get a pattern like this:

T=0: key A requested, A is not cached, start fetch #1 of A
T=1: key A requested, A is not cached, start fetch #2 of A
T=2: key A requested, A is not cached, start fetch #3 of A
T=3: key A requested, A is not cached, start fetch #4 of A
T=4: key A requested, A is not cached, start fetch #5 of A
T=5: fetch #1 of A completes and is added to the cache
T=6: key A requested, A is cached, return value immediately

In this example, the value for A is fetched 5 times despite the caching  
mechanism. If the fetching takes a long time compared to the rate at 
which requests are coming in, this effect gets worse at a quadratic 
rate: the total time spent fetching is the number of requests that come 
in during the fetching of the first request times the duration of the 
fetch.

To avoid this, you could put a Deferred for the fetch operation in the 
cache or in a separate dictionary and if you get another request for the 
same key before the fetch completes, return that Deferred instead of 
starting another fetch.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Need some enlightenment on using web client properly, or maybe nudge a bug to get fixed

[Maarten ter Huurne]

On Thursday, 11 July 2019 12:00:33 CEST Jarosław Fedewicz wrote:

> Is there a neat way to list all pyOpenSSL objects in a running Twisted
> program? Or maybe TCPConnection objects, since those might hook to
> the zope.interface machinery?

Not specific to Twisted, but you can get a list of all objects tracked 
by the garbage collector using "gc.get_objects()" and then filter that 
by class.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] finding out what's blocking the reactor

[Maarten ter Huurne]

On Wednesday, 15 May 2019 13:11:29 CEST Scott, Barry wrote:

> In my experience gc is low level noise only.
> Our work load is very heavy and its never been an issue.

It depends entirely on how many objects you have. With a few GBs worth 
of relatively small objects, a full garbage collection cycle can take 
over a minute. But in normal circumstances, the overhead indeed isn't 
relevant.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] finding out what's blocking the reactor

[Maarten ter Huurne]

On Wednesday, 15 May 2019 07:30:54 CEST Chris Withers wrote:
> On 14/05/2019 22:14, Werner Thie wrote:
> > Hi all
> > 
> > with periods of seemingly no activity the gc comes to mind.
> 
> Hmm, gc blocking the process is an interesting thought, what evidence
> would you see with gdb, etc, to show that this was the problem?

A simple way is to call gc.disable() and see if the problem goes away.

Note that gc.disable() only disables the mark-and-sweep part of garbage 
collection, which is the part that can cause the Python interpreter to 
freeze. It doesn't disable garbage collection when the refcount of an 
object reaches zero.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Drop support for Python 3.4?

[Maarten ter Huurne]

On Wednesday, 27 March 2019 06:04:17 CET Glyph wrote:
> > On Mar 26, 2019, at 7:59 PM, Craig Rodrigues
> >  wrote:
> > 
> > What do people think of dropping Twisted support for Python 3.4?
> > 
> > According to https://devguide.python.org/#status-of-python-branches
> > 
> > 
> > Python 3.4 EOL'd on March 19, 2019.
> > 
> > In the Python 3 world, we have Python 3.4, 3.5, 3.6, 3.7, and at
> > the end of this year we will have Python 3.8.
> > 
> > That's quite a lot of Python versions to support.
> > 
> > Python 3.5 introduced async/await keywords, which are very relevant
> > to Twisted:
> > https://docs.python.org/3.5/whatsnew/3.5.html#whatsnew-pep-492
> > 
> > 
> > If it makes sense, it would be nice to use these keywords as
> > first-level features in Twisted.
> > 
> > Since Amber brought up discussion of dropping Python 2.7 here:
> > https://twistedmatrix.com/pipermail/twisted-python/2019-March/032234
> > .html
> >  > 34.html>
> > 
> > I thought I would raise dropping Python 3.4 also.
> 
> I'll let any 3.4 users speak for themselves if they're out there, but
> while I can imagine a host of reasons we might want to still support
> 2.7, I can't think of any that we'd want to hang on to 3.4 any longer
> than necessary.  3.5 still has the lingering benefit of a
> production(-ish) pypy, so we might not want to jump to 3.6-only
> anyway, but if it's unsupported by python core, let's get rid of it. 
> Faster round trips through CI are reason enough :-).

Python 3.5 is still the default python3 version in Debian stable, so 
updating beyond that might make it more complex for Debian users to 
install Twisted.

One advantage of 3.5 over 3.4 is that support for type annotations is 
included (with some limitations compared to 3.6). Is this something that 
Twisted plans to adopt?

My personal experience with type annotations so far as that they 
occasionally uncover a bug, but the main benefit is in having formal 
documentation of types that can be verified by tooling (mypy).

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] A Proposal for reducing the burden of developing on Twisted by dropping Python 2 support

[Maarten ter Huurne]

On Monday, 25 March 2019 11:15:22 CET Amber Brown wrote:

> One of my rationales is that from some analysis of PyPI download
> statistics, the vast majority of Python 2 users are using old versions
> of Twisted, while nearly all our Python 3 users are on the latest
> version. As such, I believe freezing a version that will get security
> updates but no new features would not be a massive loss to those
> stuck on Python 2 for whatever reason.

I think this makes sense: applications that are still gaining new 
features should be on Python 3 by now. For applications on Python 2 the 
main concern would be to keep them running while the world around them 
updates (new OS versions etc).

> Comments are welcome, as well as which timeline seems reasonable.

Python 2.7 support will end early 2020. In my opinion it is unnecessary 
to support Twisted on Python 2.7 for longer than Python 2.7 itself is 
supported. So all of the proposed timelines are generous, in my opinion.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Thread Consumption Problem in Daemon?

[Maarten ter Huurne]

On maandag 19 november 2018 12:40:20 CET Darren Govoni wrote:
> Hi,
>   I am using twisted to run my Flask app via WSGI like so.
> 
> twistd --pidfile $PORT/pidfile -l $PORT/logfile  -n web --port
> tcp:$PORT --wsgi my.app
> 
> Naturally, I have functions representing routes that enter and exit
> just fine.
> 
> However, I notice the twisted daemon process is :"gathering threads".
> Eventually system runs out of them.
> 
>  Here's a full status for one twisted server. 504 threads???

I have a server running inside twistd which uses exactly 1 thread after 
running for a few weeks, so the problem may not be in twistd itself.

I'm using a reverse-proxy HTTP setup though, not WSGI. Maybe the problem 
is specific to WSGI, Flask or your application?

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Using twistd with -c option causes permission error

[Maarten ter Huurne]

On Tuesday, August 28, 2018 2:10:22 AM CEST Richard Shea wrote:
> I'm trying to use the -c option of twistd like this :
> 
> twistd web --wsgi bar.app -c foo.cer  -k privkey.pem --https=4433
> 
> I'm pointing it at a cert with perms like this "-rw-r--r-- 1 root root" but
> twistd complains about a permission error .
> 
> I'm puzzled ... surely twistd only needs to read that file ?

Is it complaining about the permissions on the cert or on the private key? 
Some applications (like SSH) reject private keys if they are world-readable, 
as a precaution.

Bye,
Maarten



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Deferred getChild (#3621)

[Maarten ter Huurne]

On Monday 30 May 2011, Laurens Van Houtven wrote:

 My use case for this is txYoga https://github.com/lvh/txyoga. Long
 story short, it lets you write REST-y webapps. So, you could have
 something like:
 
 http://www.twistedmatrix.com/labs/glyph
 
 In local jargon, labs is a collection, and glyph is an element in it.
 Right now, the only collection that actually exists is an in-memory one,
 and the next one I'm going to write is a SQLite one, so blocking on
 child access seems relatively reasonable. In general, however, this
 should be implemented using Deferreds, since the information about glyph
 might be stored in a database far, far away.

A possible alternative would be to create a child resource that remembers 
the child name and do the database lookup in the render method. If there is 
a matching record, render a page with its data, otherwise render a 404 page.

I cannot predict whether the resulting code would be clean or ugly though, 
so I don't know how feasible this approach is.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] CentOS error installing

[Maarten ter Huurne]

On Wednesday 18 August 2010, Landreville wrote:
 Hello,
 
 I'm running CentOS 5.5 64bit and when I tried to install twisted using
 easy_install I get this error message:
 
 Running Twisted-10.1.0/setup.py -q bdist_egg --dist-dir
 /tmp/easy_install-tgKRDq/Twisted-10.1.0/egg-dist-tmp-wd-5Ri
 twisted/runner/portmap.c:10:20: error: Python.h: No such file or
 directory

Is it possible you don't have the Python headers installed? They are 
probably in a package named python-dev, python-devel or something 
similar.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Punching a firewall with Twisted?

[Maarten ter Huurne]

On Thursday 17 June 2010, Stefan Reich wrote:

 Yeah, I found that too... it looks quite interesting, but as you say, it
 is UDP-based. I know that there is a working TCP-solution. As I said,
 Deluge does it, and for example, the original Bittorrent client does it
 too.

Maybe it uses UPNP to tell the router which ports to open?

As far as I know, there are no hole-punching techniques for TCP like there 
are for UDP. So the only way to get a port forward is to configure the 
router, manually or through an API like UPNP.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted easy bugs, and other thoughts about getting people involved in Twisted

[Maarten ter Huurne]

On Wednesday 17 March 2010, Asheesh Laroia wrote:

 Then it actually scrapes the Trac-rendered HTML because the CSV export
 (as far as I know) can't tell us answers to questions like, How many
 people were active on this bug? I find that's a useful thing to show
 off.

Not sure if it helps in your case, but I once made a script that produced a 
table of tickets associated with a milestone, to include with release notes. 
It needed the 1-line summary, which was not included in the CSV export. I 
thought I would have to resort to scraping the HTML, but then found that the 
RSS feed contained all the info I needed.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Escaping variable names

[Maarten ter Huurne]

On Friday 05 March 2010, Kamil Wasilewski wrote:
  server.doGetCountries(**{
  'country-code': COUNTRYID, 'webapi-key': WEBAPIKEY
  })
 
   The above gives a syntax error...

What kind of syntax error? I tested this approached on the interactive 
Python shell and it worked fine there:

 def f(**kvargs):
...   print kvargs
... 
 f(a = 1, **{'b-c': 12})
{'a': 1, 'b-c': 12}

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Does the default session expiry work?

[Maarten ter Huurne]

On Tuesday 02 March 2010, markscottwri...@gmail.com wrote:

 I'm using twisted 8.2 and (after reading glyph's latest Twisted in 60
 seconds entry), I've realized that my sessions never expire. I never
 added any session expiration code, but I thought that the default code
 expires after 15 minutes.
 
 In the below example, I modified twisted\web\server to make
 Session.sessionTimeout be 30 seconds (I also put a print statement in
 Session's ctor, as the log shows), then did two GETs separated by more
 than 30 seconds. As the log shows, it looks like the exact same session
 uid is being returned. I would expect it to be deleted by
 Session.expire, since time - lastModifiedTime  30. Am I missing
 something?

Maybe you are encountering bug 3458?

http://twistedmatrix.com/trac/ticket/3458

If so, upgrading Twisted would help.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] debugging a memory leak

[Maarten ter Huurne]

On Tuesday 23 February 2010, Alec Matusis wrote:

 When I start the process, both python object sizes and their counts rise
 proportionally to the numbers of reconnected clients, and then they
 stabilize after all clients have reconnected.
 At that moment, the external RSS process size is about 260MB. The
 internal size of all python objects reported by Heapy is about 150MB.
 After two days, the internal sizes/counts stay the same, but the external
 size grows to 1500MB.
 
 Python object counts/total sizes are measured from the manhole.
 Is this sufficient to conclude that this is a C memory leak in one of the
 external modules or in the Python interpreter itself?

In general, there are other reasons why heap size and RSS size do not match:
1. pages are empty but not returned to the OS
2. pages cannot be returned to the OS because they are not completely empty

It seems Python has different allocators for small and large objects:
http://www.mail-archive.com/python-l...@python.org/msg256116.html
http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-
a-large-object.htm

Assuming Python uses malloc for all its allocations (does it?), it is the 
malloc implementation that determines whether empty pages are returned to 
the OS. Under Linux with glibc (your system?), empty pages are returned, so 
there reason 1 does not apply.

Depending on the allocation behaviour of Python, the pages may not be empty 
though, so reason 2 is a likely suspect.

Python extensions written in C could also leak or fragment memory. Are you 
using any extensions that are not pure Python?

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Sending jpeg data over TCP/IP

[Maarten ter Huurne]

On Sunday 14 February 2010, Alexandre Quessy wrote:

 This said, sending them using a programmer's solution - not a sysadmin
 solution - would be closer to my own skills, so I am interested in
 knowing if this could be suitable. I think, though, that it would be
 faster to use a transfert protocol that would be implemented in C, not
 Python. Am I wrong?

I would suggest to implement it in Python first and then benchmark it. Maybe 
the simplest implementation is already fast enough. Maybe the bottleneck is 
the network or the disk you're writing to; in that case you would be better 
off upgrading your switches or buying an SSD instead of writing C code.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted webserver performance issues

[Maarten ter Huurne]

On Sunday 08 November 2009, James Y Knight wrote:

 When I last looked into the performance issues, I found that sometimes
 trac appears to block for long periods of time without releasing the
 GIL. That seems to be the core of the performance issues, currently.
 When it's responding normally, it's perfectly snappy. But, sometimes,
 it blocks for 10sec at a time.

How much memory does Trac use? Python garbage collection works in 
generations, which leads to recently allocated objects to be checked for 
garbage more often than older objects.

http://docs.python.org/library/gc.html

When collecting the oldest generation (2), we saw the Python VM freeze for 
several minutes. This was on a process that had several GB of allocated 
memory though. There was enough RAM to contain all data, but it's still 
possible the OS (Linux) swapped out pages there were not used for a long 
time to get more memory for caching.

We decided to work around the issue by disabling mark-and-sweep collection 
and relying on reference counting only. We had to add some code to forcably 
break reference cycles, but this was only needed for a small number cases.

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Can browse, but getPage() fails on localhost address

[Maarten ter Huurne]

On Thursday 29 October 2009, Steve Steiner (listsin) wrote:

 I can browse to either URL in any browser, getPage() works fine on the
 external address, 404's on the localhost address.

Does the same thing happen with 127.0.0.1 instead of localhost in the 
URL? I once encountered a nasty setup where a machine itself did not have 
localhost in /etc/hosts and the DNS server returned the IP address of 
another machine on the network...

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Gridspy - a new project using Twisted

[Maarten ter Huurne]

On Monday 26 October 2009, Tom Leys wrote:

 Gridspy provides you with an interactive view of resource usage in your
 building. It gives you hard data on your consumption patterns and helps
 you to make informed decisions.

How can other people get their own Gridspy? Do you plan to release it as a 
do-it-yourself project or as a commercial product?

Bye,
Maarten

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] inlineCallbacksDecorator

[Maarten ter Huurne]

On Sunday 07 June 2009, Terry Jones wrote:

 So I wrote a decorator specifically for inlineCallbacks decorated
 functions:

 from twisted.internet import defer

 def inlineCallbacksDecorator(callback, errback=defer.passthru):
 def wrap(f):
 def wrapper(*args, **kw):
 d = f(*args, **kw)
 if isinstance(d, defer.Deferred):
 return d.addCallbacks(callback, errback,
   callbackArgs=args,
 callbackKeywords=kw, errbackArgs=args, errbackKeywords=kw)
 # We were used to decorate a non-inlineCallbacks function.
 raise Exception(
 'Function %r was decorated with
 inlineCallbacksDecorator but '
 'did not return a deferred (did you forget
 to also decorate '
 'with inlineCallbacks?)' % f.__name__)
 return mergeFunctionMetadata(f, wrapper)
 return wrap

 You can use it like this:

 def _cbok(result, a, b=None):
 print 'In _cbok: a=%r b=%r' % (a, b)
 return result

 def _cberr(failure, *args, **kw):
 pass
 # Do something  and maybe also return the failure.

 @inlineCallbacksDecorator(_cbok, _cberr)
 @defer.inlineCallbacks
 def frog(a, b=None):
 print 'a = %r, b = %r' % (a, b)
 result = yield produceDeferred()
 # yield produceErr()
 defer.returnValue(a)

As far as I can see, this decorator could be applied to any function that 
returns a Deferred, not just to inlineCallbacks.

Bye,
Maarten


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] high server loads on FreeBSD - does kqueue reactor help?

[Maarten ter Huurne]

On Tuesday 20 January 2009, Werner Thie wrote:

 With a four core, four twisted http server processes (nevow/athena on
 top), dual 100 MBps interfaces, all four cores showing CPU loads of 50%
   plus, I observe the following degradation in the network traffic with
 tcpdump:

 - 'TCP checksum incorrect' raising to almost 25% of all packets observed
   from one client IP
 - TCP Ack'ed lost segment raising to 0.5% of all packets observed

Incorrect checksums sounds like a hardware problem. Or possibly a driver 
problem if the checksum calculation is done by the CPU instead of the 
network card. In any case it is not possible for user space code like 
Twisted to generate incorrect TCP checksums.

Bye,
Maarten


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] monitoring net resources with twisted

[Maarten ter Huurne]

On Thursday 04 December 2008, Sidharth Jain wrote:
 I want to track a set network resources like Linux servers,hosts for
 network connectivity , and want to keep track of their activeness and
 possible inactivity due to network outage or any other reason. The
 hosts/serves I'm trying to monitor are regular Linux boxes with 2.6.x
 kernel. My personal preference would be a dynamic background heart-beat
 process which could sit on these servers/hosts and send periodic
 udp-packets to a master server to signal host activity and inactivity.

You could certainly write that with Twisted, but why not adopt an existing 
monitoring application like Nagios or Zenoss?

Bye,
Maarten


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python