Hi Janos
Thanks for your feedback. I think in this case, perhaps a short
term win would be to split out the global configuration data from
the storage metadata. Once we do this, it may be possible to push
around different configurations and try out some HA alternatives.
>in this case be aware that all emails for a single customer (=domain)
>must fit to a single piler node, so you have to size the nodes (disk,
>memory, etc) carefully. If you can add resources to the given node,
>then it's probably OK
You may be right, but then this is inevitable in the event that client
data is distributed across all nodes. Sooner or later, you will run
into capacity limitations on individual nodes whether you serve one
client from a node or not - in which case resources would need to be
extended. One way to extend this would be to have the capability of
adding additional storage volumes when existing storage is nearing
capacity.
/var/piler/store/san1/ 90%used
/var/piler/store/san1/client1/00/2013/dec/...
/var/piler/store/san2/ 1% used
/var/piler/store/san2/client1/00/2014/jan/ ->...
The advantage of the apparently restrictive configuration I described
in the Extremeshock example, is that it allows cluster groups to be
created which utilise Piler mostly in it's current form with fewer
modifications than would be needed to support a larger federated
cluster model.
Btw, I like your "unlimited" scaling idea. I strongly believe this
is going to be a requirement of future appliances and clusters a
few years from now. If you really want to design a new archiving
solution around this model, it will pay off in the long term. This
model lends itself well to selling Piler as a clustered hardware
appliance.
However, as you said, we still have to meet short term goals until
then.
Regards
----- Original Message -----
From: "Janos SUTO" <[email protected]>
To: [email protected]
Sent: Wednesday, 26 February, 2014 3:28:19 PM
Subject: Re: piler scalability
Hello Darryl,
thanks for your comments.
On 2014-02-25 08:18, Darryl Sutherland wrote:
>
> Also several more observations. I agree with the previous post
> regarding
> large MySql replications - a busy db such as a large Mailpiler
> implementation
> is likely to get behind and out of sync due to sheer volume of
> records. I'm
> all for replicating a db for availability, but that should be mostly
> static
> configuration data such as SaaS clients, ui preferences and HA node
> maps.
> Metadata should remain local to the node servers for performance.
ok, can be done, however we need to find a way for the gui to query
more worker
nodes' (where piler runs and stores data) databases. One possible
solutions
could be using the FEDERATED engine
Btw. the gui. Yesterday I tested hiphop (facebook's approach to improve
php
performance), and at least the admin panel worked with it. I encourage
all
piler users to try it. One small fix is required in
system/database/mysql.php
in the function called "query". Change the following line
$s->execute($arr);
to
try {
$s->execute($arr);
}
catch(PDOException $exception) { }
You may use hiphop with nginx fastcgi, eg. on 127.0.0.1:9000
hhvm --mode server -vServer.Type=fastcgi -vServer.Port=9000
-vServer.IP=127.0.0.1
Note that the hiphop stuff requires a 64-bit OS.
> Also, to achieve HA, smtp mail can be routed to more than one server
> per client
> - 2 or three servers in a cluster depending on replication
> preference. So if I
> have 9 servers, I may choose to bcc inbound smtp replica copies
> across 3 of them
> for a given client to achieve performance and availability. Because
> each of the
> servers processes mail independently of each other, if any one of the
> servers
> goes down, or corrupts a db, I can remove it from the cluster and the
> remaining
> one or two will still continue to serve out of their own metadata
> db's. Sphinx
> indexing should also be per node independent in the same way. All the
> frontend
> ui needs to do is lookup a client in the HA map and proxy piler to
> any of the
> replicated piler nodes - perhaps using a weighted round robin table
> in the
> configuration DB.
I have something similar in mind. Create 2 (or even more) groups, eg.
groupA and
groupB. A group represents some piler worker nodes (piler + sphinx +
mysql), and
a given node is responsible for a set (or shard) of data, eg. one
year's data on
one worker node, etc. Then it's the GUI's responsibility to assemble
the search
result from the appropriate worker nodes.
By using smtp frontends it's trivial to distribute emails to both
groups, and the
groups can be located even in different datacenters, if it's required.
Partitioning is also possible based on customers, so you may put
customer1-10
to worker node #1, then customer11-20 to node #2, it's really up to
you.
By using multiple GUI nodes with multiple worker nodes (for the given
customer),
I think pretty a impressive layout can be done.
> Plan B The extremeshock option:
> To go the Extremeshock route, a robust and simpler HA cluster can be
> achieved by
> allowing every piler server to serve both webui and data, but with a
> relational map
> of their peers so that they know where to proxy clients they are not
> authoritative
> for.
>
>
> Additionally, they can have a map of replication peers for clients
> that they
> are authoritative for. So if mail is routed to node01, it knows via
> it's replica
> map that it must also bcc a copy to node3 and node5. So node01
> stores a copy and
> forwards to node03 and node05. Node03 and node05 know the mail
> arrived from node01,
> consequently they store and do not replicate.
>
> Using this model, every server is a ui, smtp and storage server -
> capable of HA
> replication among peers, and load balancing. Any of the servers can
> receive email
in this case be aware that all emails for a single customer (=domain)
must
fit to a single piler node, so you have to size the nodes (disk,
memory, etc)
carefully. If you can add resources to the given node, then it's
probably
OK.
> and bcc replicas to peers in a replication cluster. All servers know
> of other nodes
> in the Piler network, and know which hosts to proxy to for client
> data they don't
> serve. This means that inbound smtp or web traffic can load balance
> to any of the
> piler servers and there is intelligence in place on every piler node
> to ensure that
> traffic is correctly proxied or served.
>
> One thing I would still add over the above is a central configuration
> database to
> store SaaS information, HA peering/replication maps and possibly
> smtp routing tables.
> This DB stores mostly static information and can easily be
> replicated or dumped to
> backup given it's comparatively small size.
yes, it's a good idea to centrally manage these, then push configs to
the worker nodes.
Note that we have been brainstorming heavily regarding a nice and
scalable
piler. Tons of different scenarios and options arouse however I think
most
of them are outside the scope of piler, and they are choices for the
given
administrator to pick an approach.
My job is to to shape piler to be as flexible as possible to support
your
various layouts. So the initial layout I sent to the list some days ago
is
just one possible solution. If someone doesn't want a mysql cluster,
fine.
So as a conclusion for now, I'd say describe your choice of layout, and
I
can tell you whether piler fits in or not, or what to change on piler's
side.
Btw. I contacted sphinx whether I need to purchase a commercial
licence,
since I want to release a non-gpl, commercial version of sphinx. We are
still negotiating, but their answer will affect piler pricing. Stay
tuned.
During brainstorming another idea came: rewrite piler from the ground
to use hadoop + pgsql + solr to provide "unlimited" scaling. So scaling
and
fault tolerance is accomplished by simply dropping in new nodes to the
cluster.
It's possible (at least in theory), and even though it has advantages,
there
are some cons (for me) as well, eg. it takes time (much time), and it
would
be a very different product from what piler currently is.
So for the near future, I've decided to continue with piler as it is
today,
and improve it further.
Best regards,
Janos
