Hello Darryl,
thanks for your comments.
On 2014-02-25 08:18, Darryl Sutherland wrote:
Also several more observations. I agree with the previous post
regarding
large MySql replications - a busy db such as a large Mailpiler
implementation
is likely to get behind and out of sync due to sheer volume of
records. I'm
all for replicating a db for availability, but that should be mostly
static
configuration data such as SaaS clients, ui preferences and HA node
maps.
Metadata should remain local to the node servers for performance.
ok, can be done, however we need to find a way for the gui to query
more worker
nodes' (where piler runs and stores data) databases. One possible
solutions
could be using the FEDERATED engine
Btw. the gui. Yesterday I tested hiphop (facebook's approach to improve
php
performance), and at least the admin panel worked with it. I encourage
all
piler users to try it. One small fix is required in
system/database/mysql.php
in the function called "query". Change the following line
$s->execute($arr);
to
try {
$s->execute($arr);
}
catch(PDOException $exception) { }
You may use hiphop with nginx fastcgi, eg. on 127.0.0.1:9000
hhvm --mode server -vServer.Type=fastcgi -vServer.Port=9000
-vServer.IP=127.0.0.1
Note that the hiphop stuff requires a 64-bit OS.
Also, to achieve HA, smtp mail can be routed to more than one server
per client
- 2 or three servers in a cluster depending on replication
preference. So if I
have 9 servers, I may choose to bcc inbound smtp replica copies
across 3 of them
for a given client to achieve performance and availability. Because
each of the
servers processes mail independently of each other, if any one of the
servers
goes down, or corrupts a db, I can remove it from the cluster and the
remaining
one or two will still continue to serve out of their own metadata
db's. Sphinx
indexing should also be per node independent in the same way. All the
frontend
ui needs to do is lookup a client in the HA map and proxy piler to
any of the
replicated piler nodes - perhaps using a weighted round robin table
in the
configuration DB.
I have something similar in mind. Create 2 (or even more) groups, eg.
groupA and
groupB. A group represents some piler worker nodes (piler + sphinx +
mysql), and
a given node is responsible for a set (or shard) of data, eg. one
year's data on
one worker node, etc. Then it's the GUI's responsibility to assemble
the search
result from the appropriate worker nodes.
By using smtp frontends it's trivial to distribute emails to both
groups, and the
groups can be located even in different datacenters, if it's required.
Partitioning is also possible based on customers, so you may put
customer1-10
to worker node #1, then customer11-20 to node #2, it's really up to
you.
By using multiple GUI nodes with multiple worker nodes (for the given
customer),
I think pretty a impressive layout can be done.
Plan B The extremeshock option:
To go the Extremeshock route, a robust and simpler HA cluster can be
achieved by
allowing every piler server to serve both webui and data, but with a
relational map
of their peers so that they know where to proxy clients they are not
authoritative
for.
Additionally, they can have a map of replication peers for clients
that they
are authoritative for. So if mail is routed to node01, it knows via
it's replica
map that it must also bcc a copy to node3 and node5. So node01
stores a copy and
forwards to node03 and node05. Node03 and node05 know the mail
arrived from node01,
consequently they store and do not replicate.
Using this model, every server is a ui, smtp and storage server -
capable of HA
replication among peers, and load balancing. Any of the servers can
receive email
in this case be aware that all emails for a single customer (=domain)
must
fit to a single piler node, so you have to size the nodes (disk,
memory, etc)
carefully. If you can add resources to the given node, then it's
probably
OK.
and bcc replicas to peers in a replication cluster. All servers know
of other nodes
in the Piler network, and know which hosts to proxy to for client
data they don't
serve. This means that inbound smtp or web traffic can load balance
to any of the
piler servers and there is intelligence in place on every piler node
to ensure that
traffic is correctly proxied or served.
One thing I would still add over the above is a central configuration
database to
store SaaS information, HA peering/replication maps and possibly
smtp routing tables.
This DB stores mostly static information and can easily be
replicated or dumped to
backup given it's comparatively small size.
yes, it's a good idea to centrally manage these, then push configs to
the worker nodes.
Note that we have been brainstorming heavily regarding a nice and
scalable
piler. Tons of different scenarios and options arouse however I think
most
of them are outside the scope of piler, and they are choices for the
given
administrator to pick an approach.
My job is to to shape piler to be as flexible as possible to support
your
various layouts. So the initial layout I sent to the list some days ago
is
just one possible solution. If someone doesn't want a mysql cluster,
fine.
So as a conclusion for now, I'd say describe your choice of layout, and
I
can tell you whether piler fits in or not, or what to change on piler's
side.
Btw. I contacted sphinx whether I need to purchase a commercial
licence,
since I want to release a non-gpl, commercial version of sphinx. We are
still negotiating, but their answer will affect piler pricing. Stay
tuned.
During brainstorming another idea came: rewrite piler from the ground
to use hadoop + pgsql + solr to provide "unlimited" scaling. So scaling
and
fault tolerance is accomplished by simply dropping in new nodes to the
cluster.
It's possible (at least in theory), and even though it has advantages,
there
are some cons (for me) as well, eg. it takes time (much time), and it
would
be a very different product from what piler currently is.
So for the near future, I've decided to continue with piler as it is
today,
and improve it further.
Best regards,
Janos
