Re: [OpenAFS] connection timed out, how long is the timeout?

[Jose M Calhariz]

On Sun, Feb 04, 2018 at 05:21:16PM -0500, Jeffrey Altman wrote:
> On 2/4/2018 7:29 AM, Jose M Calhariz wrote:
> > I am chasing the root problem in my infra-structure of afsdb and
> > afs-fileservers.  Sometimes my afsdb loses quorum in the middle of a
> > vos operation or the Linux clients time out talking to the
> > file servers.  To help diagnose the problem I would like to know how
> > long is the timeout and if I can change the time out connections in
> > the Debian clients and for the vos operations.
> >[...]
> > The core of my infra-structure are 4 afsdb running Debian 9, and using
> > OpenAFS from Debian 1.6.20, on a shared virtualization platform.  The
> > file-servers running Debian 9 and using OpenAFS from Debian, 1.6.20,
> > are VMs in dedicated hosts for OpenAFS on top of libvirt/KVM.
> 
> Jose,
>

(...)

Thank you for your report.  I will read it with very much attention
this nigth and again tomorrow.  I am travelling from FOSDEM to home.

> 
> Jeffrey Altman
> AuriStor, Inc.

> begin:vcard
> fn:Jeffrey Altman
> n:Altman;Jeffrey
> org:AuriStor, Inc.
> adr:Suite 6B;;255 West 94Th Street;New York;New York;10025-6985;United States
> email;internet:jalt...@auristor.com
> title:Founder and CEO
> tel;work:+1-212-769-9018
> note;quoted-printable:LinkedIn: 
> https://www.linkedin.com/in/jeffreyaltman=0D=0A=
>   Skype: jeffrey.e.altman=0D=0A=
>   
> url:https://www.auristor.com/
> version:2.1
> end:vcard
> 

Kind regards
Jose M Calhariz


-- 
--

De cem favoritos dos reis, noventa e cinco foram enforcados

--Napoleão Bonaparte
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] connection timed out, how long is the timeout?

[Jeffrey Altman]

On 2/4/2018 7:54 AM, Dirk Heinrichs wrote:
> Am 04.02.2018 um 13:29 schrieb Jose M Calhariz:
> 
>> The core of my infra-structure are 4 afsdb
> 
> Wasn't it so that it's better to have an odd number of DB servers (with
> a max. of 5)?

The maximum number of ubik servers in an AFS3 cell is 20.  This is a
protocol constraint.  However, due to performance characteristics it is
unlikely that anyone could run that number of servers in a production
cell.  As the server count increases the number of messages that must be
exchanged to conduct an election, complete database synchronization
recovery, maintain quorum, and complete remote transactions.  These
messages compete with the application level requests arriving from
clients.  As the application level calls (vl, pt, ...) increase the risk
of delayed processing of disk and vote calls increases which can lead to
loss of quorum or remote transaction failures.

The reason that odd numbers of servers are preferred is because of the
failover properties.

one server - single point of failure.  outage leads to read and write
failures.

two servers - single point of failure for writes.  only the lowest ipv4
address server can be elected coordinator.  if it fails, writes are
blocked.  If it fails during a write transaction, read transactions on
the second server are blocked until the first server recovers.

three or four servers - either the first or second lowest ipv4 address
servers can be elected coordinator.  any one server can fail without
loss of write or read.

five or six servers - any of the first three lowest ipv4 address servers
can be elected coordinator.  any two servers can fail without loss of
write or read.

Although adding a fourth server increases the number of servers that can
satisfy read requests, the lack of improved resiliency to failure and
the increased risk of quorum loss makes its less desirable.


The original poster indicated that his ubik servers are virtual
machines.  The OpenAFS Rx stack throughput is limited by the clock speed
of a single processor core.  The 1.6 ubik stack is further limited by
the need to share a single processor core with all of the vote, disk and
application call processing.  As a result, anything that increases the
overhead reduces increases the risk of quorum failures.

This includes virtualization as well as the overhead imposed as a result
of Meltdown and Spectre fixes.  Meltdown and Spectre can provided a
double whammy as a result of increased overhead both within the virtual
machine and within the host's virtualization layer.

AuriStor's UBIK variant does not suffer the scaling problems of AFS3
UBIK.  AuriStor's UBIK has been successfully tested with 80 ubik servers
in a cell. This is possible because of a more efficient protocol that is
 incompatible with AFS3 UBIK and the efficiencies in AuriStor's Rx
implementation.

Jeffrey Altman
AuriStor, Inc.
<>

smime.p7s
Description: S/MIME Cryptographic Signature

Re: [OpenAFS] connection timed out, how long is the timeout?

[Jeffrey Altman]

On 2/4/2018 7:29 AM, Jose M Calhariz wrote:
> I am chasing the root problem in my infra-structure of afsdb and
> afs-fileservers.  Sometimes my afsdb loses quorum in the middle of a
> vos operation or the Linux clients time out talking to the
> file servers.  To help diagnose the problem I would like to know how
> long is the timeout and if I can change the time out connections in
> the Debian clients and for the vos operations.
>[...]
> The core of my infra-structure are 4 afsdb running Debian 9, and using
> OpenAFS from Debian 1.6.20, on a shared virtualization platform.  The
> file-servers running Debian 9 and using OpenAFS from Debian, 1.6.20,
> are VMs in dedicated hosts for OpenAFS on top of libvirt/KVM.

Jose,

There is unlikely to be a single problem but since I'm procrastinating
and curious I decided to perform some research on your cell.  This
research is the type of analysis that AuriStor performs on behalf of our
support customers.  Many of the problems you are experiencing with
OpenAFS are likely due to or exacerbated by architectural limitations
that are simply not present in AuriStorFS.

Your cell has four db servers afs01 through afs04 with associated IP
addresses that rank the servers from afs01 through afs04.  therefore
afs01 is the preferred coordinator (sync site) and if its not running
afs02 will be elected.  Given there are four servers it is not possible
for afs03 or afs04 to be elected.

There are of course multiple independent ubik database services (vl, pt,
and bu) and it is possible for quorum to exist for one and not for others.

The vl service is used to store volume location information as well as
fileserver/volserver location information.  vl entries are modified when
a fileserver restarts, when a vos command locks and unlocks an entry, or
creates, updates or deletes an entry.   Its primary consumer is the afs
client which queries volume and file server location information.

The pt service stores user and group entries.  pt entries are modified
by pts when new user entries are created, modified or deleted; and when
groups are created, modified or deleted; or when group membership
information is modified.  The primary consumer is the fileserver which
queries the pt service for user and host current protection sets each
time a client establishes an rxkad connection to the fileserver.

The vl and pt services are of course ubik services.  Therefore each
vlserver and ptserver process also offers the ubik disk and vote
services which are critical.  The vote service is used to hold
elections, distribute current database version info, and maintain
quorum.  The disk service is used to distribute the database, update the
database, and maintain database consistency.  It should be noted that
the vote service is time sensitive in that packets that are used to
request votes from peers and the responses only have a limited valid
lifetime.

Some statistics regarding your vl service.  Each server is configured
with 16 LWP threads.  afs03 and afs04 have both failed to service calls
in a timely fashion since the last restart.  If those failures were vote
or disk calls then the coordinator would mark afs03 and afs04 as
unreachable, force a recovery operation, and if both were marked down
across an election could result in lose of quorum.

Since the last restart afs01 has processed 1894352 vl transactions,
afs02 1075698 transactions, afs03 2059186 transactions, and afs04
1403592 transactions.  That will provide you some idea of the load
balancing across your cache managers. The coordinator of course is the
only one to handle write transactions; the rest are read transactions.

For the pt service the transaction counts are afs01 1818212, afs02
1619962, afs03 1554918, and afs04 1075620.  Roughly on par with the vl
service load.  Like the vl service each server has 16 LWP threads.
However, unlike the vl service the pt service is not keeping up with the
requests.  Since the last restart all four servers have failed to
service incoming calls in a timely manner thousands of times each.

The pt service failing to be responsive is a problem because it has
ripple effects on the file servers.  The longer it takes a fileserver to
query the CPS data the longer it takes to accept a new connection from a
cache manager.

The ubik services in all versions of OpenAFS prior to the 1.8 branch
have been built as LWP (cooperatively threaded) processes.  There is
only a single thread in the process that swaps context state.  The rx
threads (listener, event, ...), the vote, disk, and application (vl, pt,
bu, ...) contexts are swapped in either upon a blocking event or a
yield.  Failure of a context to yield blocks other activities including
reading packets, processing requests, etc.  Like AuriStorFS the OpenAFS
1.8 series converts the ubik services (vl, pt, bu) to native threading.
This will permit the vote and disk services and the rx threads
(listener, event,...) to operate with greater parallelism.  Unlike

Re: [OpenAFS] connection timed out, how long is the timeout?

[Jose M Calhariz]

On Sun, Feb 04, 2018 at 01:27:07PM -0600, Benjamin Kaduk wrote:
> On Sun, Feb 04, 2018 at 12:29:30PM +, Jose M Calhariz wrote:
> > 
> > Hi,
> > 
> > I am chasing the root problem in my infra-structure of afsdb and
> > afs-fileservers.  Sometimes my afsdb loses quorum in the middle of a
> 
> It is a pretty disruptive event to lose quorum; do you have any idea
> what might be responsible for that happening?

In recent times I have seen two times a "vos release" of a critical
volume to fail.  I may have wrongly interpreted the error message.  So
I past it here the last one:

Could not release lock on the VLDB entry for volume XXX
u: major synchronization error
Error in vos release command.
u: major synchronization error



> 
> > vos operation or the Linux clients time out talking to the
> > file servers.  To help diagnose the problem I would like to know how
> > long is the timeout and if I can change the time out connections in
> > the Debian clients and for the vos operations.  My plan is to increase and
> 
> The ubik election to determine quorum happens every SMALLTIME (60)
> seconds, but normally the current coordinator will retain that role
> and operations can span multiple election cycles.
> 
> Most of the timeouts involved (e.g., RX_IDLE_DEAD_TIME and
> AFS_RXDEADTIME) are also on the order of a minute.
> 
> I think you'd need to recompile in order to adjust these timeouts,
> though.  And I really would recommend tracking down why you're
> losing quorum before trying to paper over things with longer
> timeouts.

I am too chasing a second problem where a Debian OpenAFS client fail
to comunicate with the fileserver and this problem is frequent.  May I
think that this timeout is about 60 seconds?  And that I need to
recompile the client to increase or decrease the timeout?




> 
> -Ben
> 
> > decrease the timeouts in OpenAFS and other timeouts in Linux to
> > identify if I have a possible problem with the data network, iSCSI
> > network, overload on the hosts of VM, overload on the file servers or
> > other possible problem.
> > 
> > The core of my infra-structure are 4 afsdb running Debian 9, and using
> > OpenAFS from Debian 1.6.20, on a shared virtualization platform.  The
> > file-servers running Debian 9 and using OpenAFS from Debian, 1.6.20,
> > are VMs in dedicated hosts for OpenAFS on top of libvirt/KVM.
> > 
> > 
> > Kind regards
> > Jose M Calhariz
> > 
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
> 

Kind regards
Jose M Calhariz


-- 
--
.adanibober odnes enilgaT .edraugA
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] connection timed out, how long is the timeout?

[Benjamin Kaduk]

On Sun, Feb 04, 2018 at 12:29:30PM +, Jose M Calhariz wrote:
> 
> Hi,
> 
> I am chasing the root problem in my infra-structure of afsdb and
> afs-fileservers.  Sometimes my afsdb loses quorum in the middle of a

It is a pretty disruptive event to lose quorum; do you have any idea
what might be responsible for that happening?

> vos operation or the Linux clients time out talking to the
> file servers.  To help diagnose the problem I would like to know how
> long is the timeout and if I can change the time out connections in
> the Debian clients and for the vos operations.  My plan is to increase and

The ubik election to determine quorum happens every SMALLTIME (60)
seconds, but normally the current coordinator will retain that role
and operations can span multiple election cycles.

Most of the timeouts involved (e.g., RX_IDLE_DEAD_TIME and
AFS_RXDEADTIME) are also on the order of a minute.

I think you'd need to recompile in order to adjust these timeouts,
though.  And I really would recommend tracking down why you're
losing quorum before trying to paper over things with longer
timeouts.

-Ben

> decrease the timeouts in OpenAFS and other timeouts in Linux to
> identify if I have a possible problem with the data network, iSCSI
> network, overload on the hosts of VM, overload on the file servers or
> other possible problem.
> 
> The core of my infra-structure are 4 afsdb running Debian 9, and using
> OpenAFS from Debian 1.6.20, on a shared virtualization platform.  The
> file-servers running Debian 9 and using OpenAFS from Debian, 1.6.20,
> are VMs in dedicated hosts for OpenAFS on top of libvirt/KVM.
> 
> 
> Kind regards
> Jose M Calhariz
> 
> -- 
> --
> 
> A Coca-Cola encarna a verdadeira beleza do capitalismo. Ela é uma espécie de 
> religião secular, sem ensinamento moral nem outro mandamento que não seja o 
> aumento do consumo de sua bebida
> 
> --Mark Pendergrast
> ___
> OpenAFS-info mailing list
> OpenAFS-info@openafs.org
> https://lists.openafs.org/mailman/listinfo/openafs-info
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] connection timed out, how long is the timeout?

[Jose M Calhariz]

On Sun, Feb 04, 2018 at 01:54:26PM +0100, Dirk Heinrichs wrote:
> Am 04.02.2018 um 13:29 schrieb Jose M Calhariz:
> 
> > The core of my infra-structure are 4 afsdb
> 
> Wasn't it so that it's better to have an odd number of DB servers (with
> a max. of 5)?

Yes, it would be better with an odd number.  For historical reasons is
stuck on 4.  But I think this is not the root cause of my problem.

> 
> Bye...
> 
>     Dirk
> 

Kind regards
Jose M Calhariz


-- 
--

A Coca-Cola encarna a verdadeira beleza do capitalismo. Ela é uma espécie de 
religião secular, sem ensinamento moral nem outro mandamento que não seja o 
aumento do consumo de sua bebida

--Mark Pendergrast
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] connection timed out, how long is the timeout?

[Dirk Heinrichs]

Am 04.02.2018 um 13:29 schrieb Jose M Calhariz:

> The core of my infra-structure are 4 afsdb

Wasn't it so that it's better to have an odd number of DB servers (with
a max. of 5)?

Bye...

    Dirk

-- 
Dirk Heinrichs 
GPG Public Key: D01B367761B0F7CE6E6D81AAD5A2E54246986015
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de




signature.asc
Description: OpenPGP digital signature