Thanks Janos
That did it. Build completed after making the makefile change.
Regards
----- Original Message -----
From: "Janos SUTO" <[email protected]>
To: [email protected]
Cc: "Sam Gelbart" <[email protected]>
Sent: Monday, 10 February, 2014 12:05:13 PM
Subject: Re: How scaleable is mailpiler?
Hello Darryl,
try the following. Edit the top level Makefile, and fix the "SUBDIRS"
variable in line 25, so it should look like:
SUBDIRS = src etc util init.d
Then try running make again.
Janos
On 2014-02-07 15:17, Darryl Sutherland wrote:
> Hi Janos
>
> I've tried to compile the preliminary SaaS version of the Piler code
> for testing, and the build fails:
>
> ./configure --localstatedir=/var --enable-starttls
> --enable-tcpwrappers --with-database=mysql --enable-saas
>
> some configure lines......................checking for compress in
> -lz... yes
> checking for mysql_config... yes
> checking for main in -lguide... no
>
> run as user: piler
> piler data directory: /var/piler
> link mode: dynamic
>
> database: mysql
> tre library: yes
> starttls support: yes
> tcpwrappers support: yes
> SaaS: enabled
>
> pdftotext: /usr/bin/pdftotext
> catdoc: /usr/bin/catdoc
> catppt: /usr/bin/catppt
> ppthtml: /usr/local/bin/ppthtml
> xls2csv: /usr/bin/xls2csv
> unrtf: /usr/bin/unrtf
> tnef: /usr/bin/tnef
>
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: creating src/Makefile
> config.status: creating etc/Makefile
> config.status: creating util/Makefile
> config.status: creating init.d/Makefile
> config.status: creating test/Makefile
> config.status: creating contrib/imap/Makefile
> config.status: creating contrib/reload/Makefile
> config.status: creating piler-config.h
>
> So far, so good...
>
> make
>
> Some compiler output..............gcc -O2 -Wall -g -I. -I.. -I../..
> -I../../src -I../src -I/usr/include/mysql -g -static-libgcc
> -fno-omit-frame-pointer -fno-strict-aliasing -DMY_PTHREAD_FASTMUTEX=1
> -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o import import.c -lpiler
> import_helper.o -lz -lm -ldl -lcrypto -lssl -ltre -lwrap -lnsl
> -L/usr/lib64 -lmysqlclient_r -lpthread -lz -lm -lrt -lssl -lcrypto
> -ldl -L. -L../src
> import_helper.o: could not read symbols: File in wrong format
> collect2: ld returned 1 exit status
> make[1]: *** [import] Error 1
> make[1]: Leaving directory
> `/usr/local/src/piler-saas-beta-build-860/test'
> make: *** [all-recursive] Error 1
>
> Do you know what could be causing this?
>
> Regards
>
> ----- Original Message -----
> From: "Janos SUTO" <[email protected]>
> To: [email protected]
> Cc: "Sam Gelbart" <[email protected]>
> Sent: Wednesday, 5 February, 2014 1:51:44 PM
> Subject: Re: How scaleable is mailpiler?
>
> Hello Darryl,
>
> I've made a proof of concept of the improved saas features, you may
> get
> it at
> https://bitbucket.org/jsuto/piler/downloads/piler-saas-beta-build-860.tar.gz
> [1]
>
> A few notes on what it does, and how it does, and further thoughts.
>
> The installation:
> -----------------
>
> Make sure you test it in a test environment, as it needs some
> improvements.
> To compile it, add --enable-saas, then use the usual make, make
> install
> procedure.
>
> How to provision a new customer:
> --------------------------------
>
> It involves the following steps (see contrib/saas/provision.sh for an
> example):
>
> - create the customer specific store directory:
> /var/piler/store/customer1/00
> - create a separated sphinx directory: /var/piler/sphinx/customer1
> - create a mysql database for the customer, eg. customer1
>
> (note that for the sake of simplicity I recommend to use the same
> mysql
> user and password for all customer related mysql databases, thus our
> lives
> are a bit easier when we arrive at the gui)
>
> The sphinx config file unfortunately doesn't support includes, nor
> reading
> configs from a directory, so we have to produce an aggregated
> sphinx.conf.
>
> Fortunately sphinx supports an executable config file, ie. it can
> execute it,
> the read its output as a config file. To leverage this I created a
> minimal
> template (using only one main index), see contrib/saas/sphinx.conf
> and
> contrib/saas/sphinx.tmpl.
>
> Until we agree on where to put the customer list, I added the
> following
> array to sphinx.conf: @customers = ("customer1", "customer2");
> So it's perl based, but I think I'll rewrite it to be php based, and
> to
> read
> the customer list from the "piler" database.
>
> The customer_settings table could be a good candidate, however it has
> a
> single column for domain, and what if a customer has more domains?
> However
> if you we introduced a description or customer_id field, and you
> rewrited
> all smtp traffic (for the given customer) to [email protected],
> then
> it may be suitable for this purpose.
>
> I also twisted the util/indexer.*.sh scripts, just a little to support
> indexing
> different customers. To do so, execute indexer.delta.sh
> <customer_id>.
> Of
> course you have to periodically iterate over all customers. Perhaps a
> similar
> php script should provide customer ids for the indexer scripts. Or
> perhaps
> it would be better/simpler (from the point of the shell script) if we
> had
> a directory eg. /usr/local/etc/piler/saas having empty files named as
> customer_ids.
> Then the indexer could iterate over the customers on its own, no need
> for
> querying the database all the time. The trick is to create this
> filename
> when provisioning.
>
> When adding a new customer, then his index files must be initialized,
> so
> indexer must be run, eg. indexer customerid_main1
> customerid_dailydelta
> customerid_delta ...
> And perhaps reload searchd to read the new index files.
>
> The GUI:
> --------
>
> I was lost a bit why you can't or won't rewrite right into the
> documentroot.
> However I tried the following.
>
> Maintain only a single site, eg. archive.yourdomain.com, and create
> aliases
> of the customers, eg. archive.customer1domain.com,
> archive.customer2domain.com,
> etc. to this virtualhost as well. Now the configuration part.
>
> By mapping the websites to the specific customer, it's pretty easy to
> tell
> the gui to use the appropriate mysql and sphinx databases by using
> the
> following
> config-site.php snippet:
>
> $customers = array(
> 'archive.customer1domain.com' => 'customer1',
> 'archive.customer2domain.com' => 'customer2',
> ........
> );
>
> $config['CUSTOMER_ID'] = $customers[$_SERVER['HTTP_HOST']];
>
> $config['SITE_NAME'] = $_SERVER['HTTP_HOST'];
> $config['SITE_URL'] = 'http://' . $_SERVER['HTTP_HOST'] . '/';
>
> $config['SPHINX_MAIN_INDEX'] = $config['CUSTOMER_ID'] . '_main1,' .
> $config['CUSTOMER_ID'] . '_dailydelta1,' . $config['CUSTOMER_ID'] .
> '_delta1';
>
> $config['DB_DATABASE'] = $config['CUSTOMER_ID'];
>
> $config['ENABLE_SAAS'] = 1;
>
> For a more complete example see contrib/saas/config-site.php
>
> Regarding the missing features, preventing piler from world
> domination
> :-)
> please describe what sort of HA and clustering you prefer.
>
> From what you told me I learned that the smtp frontends deliver the
> emails
> to 2 archiving hosts to prevent data loss when you lose node1.
> Moreover
> these
> nodes are independent to prevent replicating any corruption to the
> other (=pair)
> node. This is possible with piler out of the box, so please describe
> how to improve
> it to meet your expectations.
>
> Best regards,
> Janos
>
> On 2014-02-03 15:13, Darryl Sutherland wrote:
> > Hi Janos
> >
> > I think we are more or less on the same page:
> >
> > _> MY POINT IS THAT PILER SOMEHOW HAS TO KNOW WHETHER THE CURRENT
> > EMAIL IS_
> > _> FOR CLIENT1 OR CLIENT2, ... THE EASIEST WAY TO DO THIS IS TO USE
> > A_
> > _> CUSTOM __EMAIL ADDRESS ON PILER'S SIDE, EG._
> >
> > _> [email protected]_
> > _> [email protected]_
> > _> [email protected]_
> >
> > _> [email protected]_
> > _> [email protected]_
> > _> [email protected]_
> >
> > That is kinda what I had in mind, yes - as long as the smtp cluster
> > knows how to route to a specific piler server for archiving. Your
> > suggestion above will do the trick
> >
> > _> AND MOREOVER THE PER CUSTOMER DATABASE NAME CAN BE "CLIENT1",_
> > _"CLIENT2", ..._
> > _> SO THERE'S NO NEED FOR A CUSTOMER - DATABASE MAPPING._
> >
> > Right again, that would work too - as long as piler ui knows to
> > provision and search individual databases for each client instance
> the
> > server archives for. The new database could be provisioned in the
> ui
> > when a new customer is created in the SaaS customer branding ui.
> Maybe
> > the new client databases could be derived from the SaaS client
> name,
> > so piler knows where to provision the client DB and where to search
> > and store client specific data.
> > Example:
> > /var/lib/mysql/mailpiler/client1
> > /var/lib/mysql/mailpiler/client2
> > etc
> >
> > _> I THINK THERE CAN BE A GLOBAL DATABASE WHERE PILER STORES
> > POLICIES, ETC._
> > _> THE PER CUSTOMER DATABASES CONTAIN THE FOLLOWING TABLES:
> > METADATA+RCPT,_
> > _> ATTACHMENT, DOMAIN, SPH_*, TAG, NOTE, TAG, OPTION, SEARCH,
> GROUP*,
> > AND_
> > _> AUDIT._
> >
> > _> NOT SURE IF THERE SHOULD BE ANY LOCAL ACCOUNT FOR CLIENTS, AND
> I_
> > _> WOULDN'T __PUT ANY ADMINISTRATOR ACCOUNT IN THE PER USER
> RELATED_
> > _> DATABASES. IT_ _ALSO __MEANS THAT CUSTOMERS CAN'T INTRODUCE ANY_
> > _> ARCHIVING OR RETENTION __POLICIES, __I BELIEVE THAT THEY SHOULD
> BE_
> > _> MAINTAINED BY YOU (IE. THE PROVIDER)._
> >
> > Whatever get's configured in the ui (customer, ldap, etc) should be
> > capable of replication - maybe this is the global policy database
> you
> > meant
> > Whatever entries are added automatically by the Piler server on
> > archiving should be in local database - no replication.
> > I agree on account data. If a client can't provide their own ldap,
> > then we would need to carry our own ldap server for shared auth The
> DB
> > is not meant for authentication, rather it is for defining
> > authentication (ldap servers, domains, groups, etc) and policy
> > configuration, and that information, as well as defined policies
> > should be capable of replication. This way, when clients are
> > provisioned, this configuration data is also provisioned on cluster
> > peers in the DR location. But message data is not shared, since
> each
> > server generates that data themselves when they receive and archive
> > email. The question is though, how to provision the DB on the
> cluster
> > peer? Maybe a call to cluster peer(s) via pilergetd?
> >
> >> _ 5) PILER SERVERS PROVIDE PER CLIENT STORAGE LOCATION_
> > _> __
> (/VAR/PILER/STORE/00/{CLIENT1,CLIENT2,CLIENT3}/YEARMONTH?/...) -
> > THIS_
> > _>__ MAKES STORAGE DEPROV EASIER SINCE CLIENT EMAIL IS IN A GIVEN
> > LOCATION_
> > _> __RATHER THAN SHARED._
> >
> > _> __GOT IT. I HAD /VAR/PILER/STORE/{CLIENT1,CLIENT2,CLIENT3}/00/
> IN
> > MIND._
> >> _The yearmonth thing is more or less implemented, it's called "12
> >> days"._
> >
> > perfect, although the "12 days" sounds misleading - did you mean
> "12
> > months"?
> >
> >> _I PLAN TO USE THE CURRENT SPHINX SCHEME (MAIN, DELTA,
> DELTA-DELTA)
> >> PER _CUSTOMER,
> >> SO THERE CAN BE /VAR/PILER/SPHINX/{CLIENT1,CLIENT2,...}
> DIRECTORIES.
> >> PROBABLY THERE
> >> SHOULD BE A SINGLE SPHINX.CONF FILE HAVING LISTING ALL CUSTOMERS
> (ON
> >> THE GIVEN PILER NODE/CLUSTER).
> >
> > Great! But this configuration needs to be capable of being
> automated
> > by the provisioning process. Perhaps an API could be called to
> handle
> > it?
> >
> > _> 7) DEDICATED SERVERS FOR CLIENTS DETERMINED BY SERVER URL -
> > POSSIBLY_
> > _> HANDLED BY NGINX BACKEND PROXY DIRECTIVES:_
> > _> HTTP://ARCHIVE.EXAMPLE.ORG/CLIENT1_TO_50/ [2] = CLUSTER1_
> > _> HTTP://ARCHIVE.EXAMPLE.ORG/CLIENT51_TO_100/ [3] = CLUSTER2_
> >
> > _> HOW ABOUT HTTP://ARCHIVE.{CLIENT1,CLIENT2,....}.ORG/ [4] ? THE
> POINT
> > COULD_
> >> BE TO USE A SINGLE PILER GUI INSTALLATION FOR ALL CLIENTS ON THE
> >> GIVEN
> > _> PILER NODE._
> >
> >> Or if you want the archive url to be http://archive.synaq.com/url,
> [5]
> > > the you can still rewrite the url to
> > > http://archive.{client1,client2,....}.org/ [6]
> > > by nginx.
> >
> >> /INDEX.PHP IS HARDWIRED AT A FEW PLACES, AND PROBABLY EVEN IN SOME
> >> CSS
> >> STUFF.
> >
> > I think I could have been clearer here:
> > On Nginx this would be http://archive [7] [1].synaq.com/client1/
> > on the backend this would be proxied to
> http://piler_cluster1/archive/ [8]
> > [2] or
> > http://piler_cluster2/archive/ [9] [2], etc
> >
> > The url location on Piler can always be the same, since piler
> doesn't
> > really care.
> > As long as it's served from a url. When proxying a url to a server
> > root, the piler
> > server will get passed the url as well, which results in a 404.
> > Instead of being
> > hardwired, can the server not rather honor the $config['SITE_URL']
> > directive?
> >
> > So if I set $config['SITE_URL'] =
> > 'http://arc-cluster1.synaq.com/archive/', this url is
> > applied everywhere? This should then not require any special
> rewrites
> > in Nginx
> > since http://archive.synaq.com/blah/ [10] [3] will always be
> cleanly mapped
> > to
> > http://{cluster-server}.synaq.com/archive/ [11]
> >
> >> DROPPING A TABLE OR DB ON CLIENT DEPROV IS DEFINITELY QUICKER AND
> >> CLEANER FROM A PERFORMANCE POINT OF VIEW THAN TRYING TO RUN A
> DELETE
> > > ACROSS A WHOLE SHARED DATABASE.
> >
> > > INDEED. WILL YOU RUN AN SQL SERVER ON EACH PILER NODE, OR DO YOU
> > PREFER
> >> TO HAVE A DEDICATED MYSQL CLUSTER TO HOLD ALL CLIENTS' PILER
> RELATED
> > > DATA?
> >
> > I think if the configuration data can be split out from the message
> > metadata,
> > the config or policy SQL data could be stored anywhere - local or
> > central, as long
> > as it can be replicated with a peer. However, because in this model
> > the archive
> > cluster servers archive independently of each other, message
> metadata
> > should
> > be on each of the piler nodes. I specifically want the cluster
> servers
> > to do
> > independent journalling and storage processing because I want to
> > minimise the
> > risk of corruption being replicated. Storage data is changing all
> of
> > the time,
> > whereas config and policy data is fairly static and could be safely
> > backed up with
> > mysqldump or other backup tools.
> >
> > Best regards,
> > Janos
> >
> >
> > Links:
> > ------
> > [1] http://archive [7]
> > [2] http://piler_cluster1/archive/ [8]
> > [3] http://archive.synaq.com/blah/ [10]
>
> Links:
> ------
> [1]
> https://bitbucket.org/jsuto/piler/downloads/piler-saas-beta-build-860.tar.gz
> [2] HTTP://ARCHIVE.EXAMPLE.ORG/CLIENT1_TO_50/
> [3] HTTP://ARCHIVE.EXAMPLE.ORG/CLIENT51_TO_100/
> [4] HTTP://ARCHIVE.%7BCLIENT1,CLIENT2,....%7D.ORG/
> [5] http://archive.synaq.com/url,
> [6] http://archive.%7Bclient1,client2,....%7D.org/
> [7] http://archive
> [8] http://piler_cluster1/archive/
> [9] http://piler_cluster2/archive/
> [10] http://archive.synaq.com/blah/
> [11] http://%7Bcluster-server%7D.synaq.com/archive/
