Hi Janos
Also several more observations. I agree with the previous post regarding
large MySql replications - a busy db such as a large Mailpiler implementation
is likely to get behind and out of sync due to sheer volume of records. I'm
all for replicating a db for availability, but that should be mostly static
configuration data such as SaaS clients, ui preferences and HA node maps.
Metadata should remain local to the node servers for performance.
Also, to achieve HA, smtp mail can be routed to more than one server per client
- 2 or three servers in a cluster depending on replication preference. So if I
have 9 servers, I may choose to bcc inbound smtp replica copies across 3 of them
for a given client to achieve performance and availability. Because each of the
servers processes mail independently of each other, if any one of the servers
goes down, or corrupts a db, I can remove it from the cluster and the remaining
one or two will still continue to serve out of their own metadata db's. Sphinx
indexing should also be per node independent in the same way. All the frontend
ui needs to do is lookup a client in the HA map and proxy piler to any of the
replicated piler nodes - perhaps using a weighted round robin table in the
configuration DB.
We use exim for our smtp filtering, and find it trivial to do per client smart
routing using a MySql lookup table.
Plan B The extremeshock option:
To go the Extremeshock route, a robust and simpler HA cluster can be achieved by
allowing every piler server to serve both webui and data, but with a relational map
of their peers so that they know where to proxy clients they are not authoritative
for.
Additionally, they can have a map of replication peers for clients that they
are authoritative for. So if mail is routed to node01, it knows via it's replica
map that it must also bcc a copy to node3 and node5. So node01 stores a copy and
forwards to node03 and node05. Node03 and node05 know the mail arrived from node01,
consequently they store and do not replicate.
Using this model, every server is a ui, smtp and storage server - capable of HA
replication among peers, and load balancing. Any of the servers can receive email
and bcc replicas to peers in a replication cluster. All servers know of other nodes
in the Piler network, and know which hosts to proxy to for client data they don't
serve. This means that inbound smtp or web traffic can load balance to any of the
piler servers and there is intelligence in place on every piler node to ensure that
traffic is correctly proxied or served.
One thing I would still add over the above is a central configuration database to
store SaaS information, HA peering/replication maps and possibly smtp routing tables.
This DB stores mostly static information and can easily be replicated or dumped to
backup given it's comparatively small size.
Regards
|
Darryl Sutherland Senior Linux Administrator |
|||||||||
 |
|
||||||||
PS: Download our free whitepaper on email archiving best practices here!
Support: +27 (0)11 262 3628 | +27 (0)861 0 SYNAQ
Block D, Sandhavon Office Park, Pongola Crescent, Eastgate Ext. 4, Sandton
----- Original Message -----
From: "Janos SUTO" <[email protected]>
To: [email protected]
Sent: Monday, 24 February, 2014 5:43:37 PM
Subject: Re: piler scalability
Hello,
thanks for your comments. However a few questions popped in my mind.
On 2014-02-24 15:35, admin-at-extremeshok-dot-com wrote:
>
>
> Maybe do a more isolated management node (gui) with multiple
> independent worker nodes (piler+searchd), which are centrally
> controlled from the management node.
>
> The worker nodes would get a map xml from the management node,
> mapping what email domains they would be storing on each worker node.
>
> It would be a good idea to provide some sort of message transport
> system between the nodes (rabbitmq) in order for them to transfer
> emails from one node to another.
since we deal with smtp, I'd rather replace rabbitmq with an smtp
frontend, so we don't have to introduce yet another protocol and
a way different type of technology.
So by tweaking your scheme we would have the following:
mail server -> smtp proxy (or frontend) #1 -> piler01,piler02,....
frontend #1 handles AAA.com,BBB.com,CCC.com and DDD.com, and it has a
transport
map to forward the given domain's emails to the appropriate piler node,
eg.
AAA.com -> piler01
BBB.com -> piler02
CCC.com -> piler03
DDD.com -> piler04
if piler02 were down, then the frontend would store and queue up those
messages
until piler02 comes back. Note that the smtp proxy could be installed
on the piler
worker nodes provided that piler is bound to 127.0.0.1, but in this
case you need
a more complex MX settings perhaps to push emails from the mail server
to the
appropriate piler node. So I'd recommend a single (pair) of postfix
node(s)
to store the transport map, and route messages to their destination
worker nodes.
However what happens if piler02 is vaporized? Then what if AAA.com is
so huge
that it barely fits in a single piler node? Your only choice is to
scale up (ie.
add more cpu, memory, disk to the given mode).
With such a setup you also need a http proxy to forward AAA.com users
to piler01,
BBB.com users to piler02, etc.
Janos
> Now my thought:
>
> Single Instance
> ==============
> [master gui + piler01]
> XML : AAA.com, BBB.com,CCC.com,DDD.com
>
> Failover + scaling
> =========================
> [master gui + piler01] --- [slave gui + piler02]
> XML : AAA.com , BBB.com | CCC.com, DDD.com
>
> Failover + Scaling
> =====================
> GUI/control: [master gui] <==> [slave gui] (optional slave)
> \\\\ the master gui, could be located on piler01, to allow for a
> single instance
> Worker nodes: [piler01] --- [piler02]---[piler03]---[piler04]
> XML : AAA.com | BBB.com | CCC.com | DDD.com
>
> Treat the worker nodes as independent nodes, ie. storage and
> searchd/database is on each node.
>
> piler01 is archive all email for AAA.com, piler02 is archiving all
> email for BBB.com
>
> Normal operation, email from AAA.com arrives, piler01 process and
> archives
> Infinite scaling would be possible
> BBB.com was running on piler01, and then moved to piler02 to scale..
>
> if piler02 goes offline, piler01 would know what domains belong to
> piler02, and thus would accept the messages, but not process them.
> Instead when piler02 comes online, it would send the messages to
> piler02 via rabbitmq.
>
> The gui/control could be entirely XML or Mysql based. easiest would
> be to use mysql replication, and the slave node would be readonly,
> unless it was changed to a master node.
>
> With my way, its the easiest to scale for 1----100 nodes, you could
> add HA to the gui node from the get go, or just run piler as a single
> server install.
>
> As for distributed/shared storage.. Ideal is to leave that to the
> admin, one could use DAS/NAS/ceph/glusterfs
>
>
> Regards
>
> admin-at-extremeshok-dot-com
