Hi Toad!
First: thx for your work and don't give up the NLM!
First impressions were overall positiv, when I started activating NLM on
this node about 138x. My in/out+HTL success rates went up noticeable.
Backoffs stayed low. Only thread usage raised a bit +20-30%.
Serveral days and versions
Am Mittwoch, 31. August 2011, 13:25:35 schrieb Matthew Toseland:
> On Tuesday 30 Aug 2011 21:02:54 Arne Babenhauserheide wrote:
> > what this means: if a SSK has a mean success rate of 0.16, then using
> > 0.25 as value makes sure that 95% of the possible cases don?t exhaust
> > the bandwidth. We
On Tuesday 30 Aug 2011 18:27:03 Ian Clarke wrote:
> On Tue, Aug 30, 2011 at 11:49 AM, Robert Hailey
> wrote:
>
> > On 2011/08/29 (Aug), at 12:58 PM, Ian Clarke wrote:
> >
> > The problem is that we come up with solutions that are too complicated to
> > analyze or fix when they don't work
> >
On Tuesday 30 Aug 2011 21:02:54 Arne Babenhauserheide wrote:
> Am Dienstag, 30. August 2011, 01:08:16 schrieb Arne Babenhauserheide:
> > 5) solution: count each SSK as only
> > average_SSK_success_rate * data_to_transfer_on_success.
>
> Some more data:
>
> chances of having at least this
On Tuesday 30 Aug 2011 21:02:54 Arne Babenhauserheide wrote:
Am Dienstag, 30. August 2011, 01:08:16 schrieb Arne Babenhauserheide:
5) solution: count each SSK as only
average_SSK_success_rate * data_to_transfer_on_success.
Some more data:
chances of having at least this many
On Monday 29 Aug 2011 20:34:01 Ian Clarke wrote:
On Mon, Aug 29, 2011 at 2:21 PM, Matthew Toseland t...@amphibian.dyndns.org
wrote:
- What is the average load reported by responses this node
forwarded, per
remote node
Ahhh, this one could be interesting - you could use
On Tuesday 30 Aug 2011 18:27:03 Ian Clarke wrote:
On Tue, Aug 30, 2011 at 11:49 AM, Robert Hailey
rob...@freenetproject.orgwrote:
On 2011/08/29 (Aug), at 12:58 PM, Ian Clarke wrote:
The problem is that we come up with solutions that are too complicated to
analyze or fix when they don't
Am Mittwoch, 31. August 2011, 13:25:35 schrieb Matthew Toseland:
On Tuesday 30 Aug 2011 21:02:54 Arne Babenhauserheide wrote:
what this means: if a SSK has a mean success rate of 0.16, then using
0.25 as value makes sure that 95% of the possible cases don’t exhaust
the bandwidth. We then
Am Dienstag, 30. August 2011, 01:08:16 schrieb Arne Babenhauserheide:
> 5) solution: count each SSK as only
> average_SSK_success_rate * data_to_transfer_on_success.
Some more data:
chances of having at least this many successful transfers for 40 SSKs with a
mean success rate of 16%:
On Tue, Aug 30, 2011 at 11:49 AM, Robert Hailey
wrote:
> On 2011/08/29 (Aug), at 12:58 PM, Ian Clarke wrote:
>
> The problem is that we come up with solutions that are too complicated to
> analyze or fix when they don't work
>
> The cause is complexity, which just grows and grows as we try to
On 2011/08/30 (Aug), at 3:17 AM, Thomas Bruderer wrote:
> Thank you Ian! Good message! I am 100% behind your whole post! The
> routing must go back to a simple system!
Even tearing up the current systems for a fifo queue is destructive
without organization and a means of comparison.
In the
> The entire approach of coming up with hypotheses about what is wrong,
> building a solution based on these hypotheses (without actually
> confirming that the hypotheses are accurate) and deploying it is deja
> vu, we've been doing it for a decade, and we still haven't got load
> management
Von: "Matthew Toseland" ???
>But the other question is, can queueing ever be helpful? It can if it allows
>us to route more accurately (which NLM clearly does), and/or to run enough
>requests in parallel that the longer time taken for the request to reach its
>destination is offset. Is this
After discussing and going deeper into this, it became apparent that the
problem is not overload of queues. I?ll repeat the result I came to here (and
not in chatlog form :) ):
The problem is in the load limiter:
1) we need to use all bandwidth, because latency depends on the number of
On Tuesday 30 Aug 2011 00:18:35 Matthew Toseland wrote:
> On Tuesday 30 Aug 2011 00:08:16 Arne Babenhauserheide wrote:
> > After discussing and going deeper into this, it became apparent that the
> > problem is not overload of queues. I?ll repeat the result I came to here
> > (and
> > not in
On Tuesday 30 Aug 2011 00:08:16 Arne Babenhauserheide wrote:
> After discussing and going deeper into this, it became apparent that the
> problem is not overload of queues. I?ll repeat the result I came to here (and
> not in chatlog form :) ):
>
> The problem is in the load limiter:
>
>
>
The entire approach of coming up with hypotheses about what is wrong,
building a solution based on these hypotheses (without actually
confirming that the hypotheses are accurate) and deploying it is deja
vu, we've been doing it for a decade, and we still haven't got load
management right.
On 2011/08/30 (Aug), at 3:17 AM, Thomas Bruderer wrote:
Thank you Ian! Good message! I am 100% behind your whole post! The
routing must go back to a simple system!
Even tearing up the current systems for a fifo queue is destructive
without organization and a means of comparison.
In the
On Tue, Aug 30, 2011 at 11:49 AM, Robert Hailey
rob...@freenetproject.orgwrote:
On 2011/08/29 (Aug), at 12:58 PM, Ian Clarke wrote:
The problem is that we come up with solutions that are too complicated to
analyze or fix when they don't work
The cause is complexity, which just grows and
Am Dienstag, 30. August 2011, 01:08:16 schrieb Arne Babenhauserheide:
5) solution: count each SSK as only
average_SSK_success_rate * data_to_transfer_on_success.
Some more data:
chances of having at least this many successful transfers for 40 SSKs with a
mean success rate of 16%:
for i
On Monday 29 Aug 2011 20:32:13 Ian Clarke wrote:
> On Mon, Aug 29, 2011 at 2:09 PM, Matthew Toseland amphibian.dyndns.org
> > wrote:
>
> > On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
> > > Ok, thinking about it, here is a proposal, or rather, the beginning of a
> > > proposal. I'm
Okay, I don't understand most of that, I might be able to check the math if it
was written properly, but it looks difficult. However, as far as I can see:
- The most obvious way to increase bandwidth usage would be to increase the
timeout time for output bandwidth liability (and at the same time
Am Montag, 29. August 2011, 14:32:13 schrieb Ian Clarke:
> Yes, small tweaks have worked so well for us for the last decade, leaving us
> pretty-much where we were in 2003. No, we don't understand how the current
> system works, there is no point in trying to fix something when we don't
> even
On Monday 29 Aug 2011 20:09:58 Matthew Toseland wrote:
> On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
> > Ok, thinking about it, here is a proposal, or rather, the beginning of a
> > proposal. I'm assuming that we get rid of NLM, fair sharing, and anything
> > else intended to control load,
On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
> Ok, thinking about it, here is a proposal, or rather, the beginning of a
> proposal. I'm assuming that we get rid of NLM, fair sharing, and anything
> else intended to control load, and replace it with this. We will absolutely
> need to
On Monday 29 Aug 2011 18:58:26 Ian Clarke wrote:
> On Mon, Aug 29, 2011 at 12:37 PM, Matthew Toseland <
> toad at amphibian.dyndns.org> wrote:
>
> > That is because we do not have the time or funding to empirically test
> > hypotheses. We don't gather enough data, we don't have a huge testnet to
On Monday 29 Aug 2011 18:37:39 Matthew Toseland wrote:
> On Saturday 27 Aug 2011 21:35:58 Ian Clarke wrote:
> > Matthew,
> >
> > This makes sense from the perspective of making incremental changes, but I
> > think we need to be more drastic than that. I think we need to go back to
> > the
On Saturday 27 Aug 2011 21:35:58 Ian Clarke wrote:
> Matthew,
>
> This makes sense from the perspective of making incremental changes, but I
> think we need to be more drastic than that. I think we need to go back to
> the drawing board with load management. We need to find a solution that is
>
On Mon, Aug 29, 2011 at 2:21 PM, Matthew Toseland wrote:
> > >- What is the average load reported by responses this node
> forwarded, per
> > >remote node
> >
> > Ahhh, this one could be interesting - you could use it to penalise nodes
> which spam excessively.
>
> Actually, thinking
On Mon, Aug 29, 2011 at 2:09 PM, Matthew Toseland wrote:
> On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
> > Ok, thinking about it, here is a proposal, or rather, the beginning of a
> > proposal. I'm assuming that we get rid of NLM, fair sharing, and
> anything
> > else intended to control
Ok, thinking about it, here is a proposal, or rather, the beginning of a
proposal. I'm assuming that we get rid of NLM, fair sharing, and anything
else intended to control load, and replace it with this. We will absolutely
need to simulate this before we write a single line of code to deploy
On Mon, Aug 29, 2011 at 12:37 PM, Matthew Toseland <
toad at amphibian.dyndns.org> wrote:
> That is because we do not have the time or funding to empirically test
> hypotheses. We don't gather enough data, we don't have a huge testnet to try
> stuff on over extended periods, and so on. Most
On Saturday 27 Aug 2011 21:35:58 Ian Clarke wrote:
Matthew,
This makes sense from the perspective of making incremental changes, but I
think we need to be more drastic than that. I think we need to go back to
the drawing board with load management. We need to find a solution that is
On Monday 29 Aug 2011 18:37:39 Matthew Toseland wrote:
On Saturday 27 Aug 2011 21:35:58 Ian Clarke wrote:
Matthew,
This makes sense from the perspective of making incremental changes, but I
think we need to be more drastic than that. I think we need to go back to
the drawing board
On Mon, Aug 29, 2011 at 12:37 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
That is because we do not have the time or funding to empirically test
hypotheses. We don't gather enough data, we don't have a huge testnet to try
stuff on over extended periods, and so on. Most software
Ok, thinking about it, here is a proposal, or rather, the beginning of a
proposal. I'm assuming that we get rid of NLM, fair sharing, and anything
else intended to control load, and replace it with this. We will absolutely
need to simulate this before we write a single line of code to deploy
On Monday 29 Aug 2011 18:58:26 Ian Clarke wrote:
On Mon, Aug 29, 2011 at 12:37 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
That is because we do not have the time or funding to empirically test
hypotheses. We don't gather enough data, we don't have a huge testnet to try
stuff
On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
Ok, thinking about it, here is a proposal, or rather, the beginning of a
proposal. I'm assuming that we get rid of NLM, fair sharing, and anything
else intended to control load, and replace it with this. We will absolutely
need to simulate
On Monday 29 Aug 2011 20:09:58 Matthew Toseland wrote:
On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
Ok, thinking about it, here is a proposal, or rather, the beginning of a
proposal. I'm assuming that we get rid of NLM, fair sharing, and anything
else intended to control load, and
On Mon, Aug 29, 2011 at 2:09 PM, Matthew Toseland t...@amphibian.dyndns.org
wrote:
On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
Ok, thinking about it, here is a proposal, or rather, the beginning of a
proposal. I'm assuming that we get rid of NLM, fair sharing, and
anything
else
On Mon, Aug 29, 2011 at 2:21 PM, Matthew Toseland t...@amphibian.dyndns.org
wrote:
- What is the average load reported by responses this node
forwarded, per
remote node
Ahhh, this one could be interesting - you could use it to penalise nodes
which spam excessively.
Actually,
Am Montag, 29. August 2011, 14:32:13 schrieb Ian Clarke:
Yes, small tweaks have worked so well for us for the last decade, leaving us
pretty-much where we were in 2003. No, we don't understand how the current
system works, there is no point in trying to fix something when we don't
even know
Okay, I don't understand most of that, I might be able to check the math if it
was written properly, but it looks difficult. However, as far as I can see:
- The most obvious way to increase bandwidth usage would be to increase the
timeout time for output bandwidth liability (and at the same time
On Monday 29 Aug 2011 20:32:13 Ian Clarke wrote:
On Mon, Aug 29, 2011 at 2:09 PM, Matthew Toseland t...@amphibian.dyndns.org
wrote:
On Monday 29 Aug 2011 19:28:50 Ian Clarke wrote:
Ok, thinking about it, here is a proposal, or rather, the beginning of a
proposal. I'm assuming that
After discussing and going deeper into this, it became apparent that the
problem is not overload of queues. I’ll repeat the result I came to here (and
not in chatlog form :) ):
The problem is in the load limiter:
1) we need to use all bandwidth, because latency depends on the number of
On Tuesday 30 Aug 2011 00:08:16 Arne Babenhauserheide wrote:
After discussing and going deeper into this, it became apparent that the
problem is not overload of queues. I’ll repeat the result I came to here (and
not in chatlog form :) ):
The problem is in the load limiter:
1) we
On Tuesday 30 Aug 2011 00:18:35 Matthew Toseland wrote:
On Tuesday 30 Aug 2011 00:08:16 Arne Babenhauserheide wrote:
After discussing and going deeper into this, it became apparent that the
problem is not overload of queues. I’ll repeat the result I came to here
(and
not in chatlog
Von: Matthew Toseland t...@amphibian.dyndns.org
But the other question is, can queueing ever be helpful? It can if it allows
us to route more accurately (which NLM clearly does), and/or to run enough
requests in parallel that the longer time taken for the request to reach its
destination is
On Saturday 27 Aug 2011 15:43:53 Matthew Toseland wrote:
> After trying out New Load Management on the network and seeing rather bad
> results, we need to reconsider load management. IMHO Old Load Management (the
> current system) is still not an acceptable answer.
>
> Ideal load management
After trying out New Load Management on the network and seeing rather bad
results, we need to reconsider load management. IMHO Old Load Management (the
current system) is still not an acceptable answer.
Ideal load management would:
- PERFORMANCE: Performance is the number of requests running in
Matthew,
This makes sense from the perspective of making incremental changes, but I
think we need to be more drastic than that. I think we need to go back to
the drawing board with load management. We need to find a solution that is
simple enough to reason about, and to debug if we have
I'm glad to see that the subject of complexity has come up, and if I
can speak to in a more general way...
Complexity is insidious - you start with a simple idea, the creativity
flows and over time, you are wedded to a highly coupled, inflexible
and obscure beast that is hard to distance yourself
After trying out New Load Management on the network and seeing rather bad
results, we need to reconsider load management. IMHO Old Load Management (the
current system) is still not an acceptable answer.
Ideal load management would:
- PERFORMANCE: Performance is the number of requests running in
On Saturday 27 Aug 2011 15:43:53 Matthew Toseland wrote:
After trying out New Load Management on the network and seeing rather bad
results, we need to reconsider load management. IMHO Old Load Management (the
current system) is still not an acceptable answer.
Ideal load management would:
Matthew,
This makes sense from the perspective of making incremental changes, but I
think we need to be more drastic than that. I think we need to go back to
the drawing board with load management. We need to find a solution that is
simple enough to reason about, and to debug if we have
I'm glad to see that the subject of complexity has come up, and if I
can speak to in a more general way...
Complexity is insidious - you start with a simple idea, the creativity
flows and over time, you are wedded to a highly coupled, inflexible
and obscure beast that is hard to distance yourself
56 matches
Mail list logo
