Re: [Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[Anthony Baxter]

>>> Pavlos Christoforou wrote
> OTO an RDBMS maybe a more reliable datastore than the filesystem with
> better consistency, recovery tools etc etc.

The other bigwin for an RDBMS-based one is, of course, that you
can share it between zopes. And for me, it's easier to manage a
DBMS table than a chunk of filesystem somewhere on a machine. 

> Also check whether SQLSession caches info. It used to hit the RDBMs for
> every variable access which was very prohibitive for us, but Anthony
> mentioned that he was changing that.

As of 0.2.x it sometimes does, depending on whether you've turned on 
caching - but 0.3.0 will fix it entirely, I plan to remove the non-cached
option, and do it a whole lot smarter. 

0.2.9 will be out shortly (tonight, if I can stop this sodding PC from
crashing every 10 minutes) and will be the last of the current architecture.
0.3.0 is moving to a more sane system - one that will stop my DBA from
hurting me. :)

Anthony, who's really not a fan of PC hardware.

-- 
Anthony Baxter <[EMAIL PROTECTED]>   
It's never too late to have a happy childhood.


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

[Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[Graham Chiu]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

In article <[EMAIL PROTECTED]>, chas
<[EMAIL PROTECTED]> writes
>>Graham Chiu wrote:
>>It may be the way I designed my site, but if the database serving
>>SQLSessions had to be taken down for maintenance, my site became
>>inaccessible.
>
>Same would happen to my site. Rather than try to build around 
>that, I think I'll try to keep the DB up instead - since I won't
>have much functionality without sessions.

What I meant was that it should be easy to plug in another database
engine for the session serving since there should be nothing database
specific about the interface to SQLSession. ( As it happens, there is
:-(  )  So, taking down the database for Session management should not
take down the site.

- -- 
Regards,  Graham Chiu
gchiucompkarori.co.nz
http://www.compkarori.co.nz/index.php
Powered by Interbase and Zope

-BEGIN PGP SIGNATURE-
Version: PGPsdk version 1.7.1

iQA/AwUBOSew+7TRdIWzaLpMEQI03gCfU+VlPOn/x3xXkcJaN8Pc0fVtlH0An1YK
2pdxpkp1E9Fy5aDPUdP2qp20
=8kZ/
-END PGP SIGNATURE-

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[chas]

Thank you very much for the comprehensive and speedy reply, J.

>Let me give it a try (or just dump all of what I know).

Much more than I expected - and greatly appreciated. Thank you.

>First of all it looks like you are going to want ZEO. If you don't 
>know ZEO, get to know ZEO, it fits perfectly with what you want for 
>the frontline servers.

Wow, has it been released for public consumption yet ? 
(I've been out of the Zope-loop for a while unfortunately)
But yes, I had ZEO in mind for keeping the application 
logic on the frontline servers in sync.

>As for the database, MySQL is hands down the fastest, but certainly 
>not the most robust in terms of features. Oracle is great if you have 
>$10,000 and a full time DBA but PostgreSQL 7.0 is really your best 
>bet. You might also want to take the money you were going to spend on 
>the DB2 and put it right back into DB1 and house everything there. 
>Just get a beefy machine, RAID it up for redundancy and call it a day.

Will definitely be getting RAID for the application data.


>The other idea that I have been playing around with in terms of 
>redundancy is to build something into the Zope code for the ZSQL 
>Methods that can go into a "cached" mode if your database dies. I 
>have only begun to play with this, but in theory since, Zope can 
>cache all your common data that comes from SQL you could, in effect, 
>bring down the SQL database and have Zope continue to work with the 
>cached data. I think all that it would involve (and I am sure that I 
>am simplifying the hell out of this) is to have Zope know that the 
>SQL database is no longer there (look for an error) and just turn the 
>"Maximum time (seconds) to cache results" to unlimited until the 
>database was restored.
>
>Has anyone else thought about this or played with this idea?

I think in most cases, if the DB dies, it's game over until
it goes up again.


>Graham Chiu wrote:
>It may be the way I designed my site, but if the database serving
>SQLSessions had to be taken down for maintenance, my site became
>inaccessible.

Same would happen to my site. Rather than try to build around 
that, I think I'll try to keep the DB up instead - since I won't
have much functionality without sessions.

>Based upon this experience, I would therefore now prefer to use
>different database engines for the main database and for the session
>tables.  And since for sessions, all you really want is speed without
>any fancy SQL transactions, then MySQL for the session data seems an
>appropriate use.

Exactly what I was hoping - thank you for the confirmation :)



>Graham Chiu wrote:
>>
>>Steve Alexander wrote:
>>I seem to recall from the MySQL vs Postgres thread a while ago that
>>MySQL was fastest in mostly-reading situations, whereas Postgres was far
>>better when there is quite a lot of writing as well as reading.

Interesting. I would have thought that MySQL won in writes b/c it
doesn't have the transaction overhead. Then again, Postgres 7 seems
to have updated the ante a lot.

>>If this is true, the Postgres should be a better choice for a
>>session-data DBMS.
>
> For SQLSessions, you only write the session data once, and read multiple
> times.  So, that would suit MySQL ?

I guess it depends what you're doing with your session object.
We'll be changing the properties of the session object as the 
user traverses the site, so the session object will have to be
updated (ie. updated on the RDB).


Thank you all for your help. 

chas


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[Graham Chiu]

In article <[EMAIL PROTECTED]>, Steve Alexander  writes
>> And since for sessions, all you really want is speed without
>> any fancy SQL transactions, then MySQL for the session data seems an
>> appropriate use.
>
>I seem to recall from the MySQL vs Postgres thread a while ago that
>MySQL was fastest in mostly-reading situations, whereas Postgres was far
>better when there is quite a lot of writing as well as reading.
>
>If this is true, the Postgres should be a better choice for a
>session-data DBMS.

For SQLSessions, you only write the session data once, and read multiple
times.  So, that would suit MySQL ?

-- 
Regards,  Graham Chiu
gchiucompkarori.co.nz
http://www.compkarori.co.nz/index.php
Powered by Interbase and Zope

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[Steve Alexander]

Graham Chiu wrote:
> 
> In article , J. Atwood
> <[EMAIL PROTECTED]> writes
> >As for the database, MySQL is hands down the fastest, but certainly not the most
> >robust in terms of features. Oracle is great if you have $10,000 and a full time
> >DBA but PostgreSQL 7.0 is really your best bet. You might also want to take the
> >money you were going to spend on the DB2 and put it right back into DB1 and
> >house everything there. Just get a beefy machine, RAID it up for redundancy and
> >call it a day.


> And since for sessions, all you really want is speed without
> any fancy SQL transactions, then MySQL for the session data seems an
> appropriate use.

I seem to recall from the MySQL vs Postgres thread a while ago that
MySQL was fastest in mostly-reading situations, whereas Postgres was far
better when there is quite a lot of writing as well as reading.

If this is true, the Postgres should be a better choice for a
session-data DBMS.

Of course, the *right* way to find out is to run a benchmark :-)

--
Steve Alexander
Software Engineer
Cat-Box limited

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[Graham Chiu]

In article , J. Atwood
<[EMAIL PROTECTED]> writes
>As for the database, MySQL is hands down the fastest, but certainly not the most 
>robust in terms of features. Oracle is great if you have $10,000 and a full time 
>DBA but PostgreSQL 7.0 is really your best bet. You might also want to take the 
>money you were going to spend on the DB2 and put it right back into DB1 and 
>house everything there. Just get a beefy machine, RAID it up for redundancy and 
>call it a day.

It may be the way I designed my site, but if the database serving
SQLSessions had to be taken down for maintenance, my site became
inaccessible.

Based upon this experience, I would therefore now prefer to use
different database engines for the main database and for the session
tables.  And since for sessions, all you really want is speed without
any fancy SQL transactions, then MySQL for the session data seems an
appropriate use.

Session tables wouldn't have to be too big as you could run ZScheduler
to remove expired sessions.

-- 
Regards,  Graham Chiu
gchiucompkarori.co.nz
http://www.compkarori.co.nz/index.php
Powered by Interbase and Zope.

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

[Zope] SQLSession vs FSSession for ultra-high scalability and speed.

[chas]

Hi Folks,

Sorry if this has been asked before, but can anybody 
advise on FSSession vs SQLSession for:

a) Speed.
b) Scalability.

Planned architecture employs multiple Zope/Apache 
frontline servers:

+-+
| Z/A |---+
+-+   |
  |
+-+   |
| Z/A |---+  +- DB1 
+-+   |  |
  |--|
+-+   |  |
| Z/A |---+  +- DB2 (optional)
+-+   |
  |
  |
  Etc, etc.

Z/A = 'front line' webservers running Zope/Apache.
  Contains application/business logic. 
  We are NOT using the ZODB for data storage.

DB1 = Data storage. eg. Oracle/Postgres

DB2 = potentially just for managing the sessions, if
  we use SQLSession. Would MySQL be best ?


Just how scalable can it really get ? We need it to at least
be able to scale to 12 million sessions per day (and substant-
-ially more writes to DB1 per day), even if we will be starting
out much more modestly than that.  Realistically, should I be 
looking at something like ATG Dynamo instead ?  

Thank you very much,

chas 

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )