Re: Server Maintenance Across Timezones (global)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Peng, Peng Tuck Kwok wrote: There's a lot of good suggestions here, maybe you could also justify maintaining a separate instance for the American customers. That would at least allow at a minimum to roll out changes specific for them, conform to their maintenance time :P. Yes I do realize it would be a replication of code in terms of releases but it is something to think about. For an application used this widely, presumably you'll need multiple physical machines, anyway. Since you require multiple machines, this replication of code will have to happen, anyway. You ought to be able to configure certain servers/clusters to provide, say, US-oriented content, while others provide content for other geographic areas. I think that's quite an elegant solution, actually. Thanks, Paul! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjXmM8ACgkQ9CaO5/Lv0PCwBgCcCwv6GiGR/8HFRULnIwPDNoM2 uqwAoJMB363isCzb9VJ3rAjK2T35dfQa =N+Tg -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Server Maintenance Across Timezones (global)
John- your approach works best for long term operational and maintenance considerations layersa properly architected and designed enterprise system has ability to interchange either the UI or DB with minimal impact to the other layers except for predefined 'interface points' which AOP folks call 'contract' Unfortunately the majority of 'J2EE enterprise systems' which start as prototypes have evolved into monolithic un-architected and un-documented tangled mess..where everyone is afraid to touch an attribute in one component of one layer which would crash the other layers does anyone remember Structured Exception Handling? Thanks, Martin __ Disclaimer and confidentiality note Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission. Date: Wed, 17 Sep 2008 16:08:19 -0700 From: [EMAIL PROTECTED] To: users@tomcat.apache.org Subject: Re: Server Maintenance Across Timezones (global) John5342 wrote: I get around my the same kinds of problems by keeping all the layers of the web app seperate so that i can swap them out one at a time and create a near seemless upgrade. The layers in my web apps are: 1 The web interface. 2. The application logic. (this may itself be several layers in itself if the app is complicated) 3. The database access layer. 4. The database. [...] Hope what i said is useful. I think it will be useful when we get to the point of redesigning the app from scratch. It's a bit tough to replace the data access layer of a large complex app that's been around for a long time though. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] _ Want to do more with Windows Live? Learn “10 hidden secrets” from Jamie. http://windowslive.com/connect/post/jamiethomson.spaces.live.com-Blog-cns!550F681DAD532637!5295.entry?ocid=TXT_TAGLM_WL_domore_092008
Re: Server Maintenance Across Timezones (global)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alan, Alan Chaney wrote: As far as the database side of it goes it seems to me that much of it is a question of making the 'live-update' a design requirement for any upgrades. You have to make it possible for the changes to the database to 'co-exist' and then update the live database independently of the application. When the new tables are ready, deploy your new application and (hopefully) go home smiling. This is the right way to do it, but it's a complete PITA to implement. :( I'd be interested to hear from people who have clustered solutions. Once again I suspect there may be problems with trying to sustain sessions across the upgrades. We don't do this (operating only in the US for now), but one way to do this is to have multiple clusters, and multiple clustered databases using replication across clusters. You remove one of the db clusters from the replication schedule and one of the app server clusters from normal service, and let your clients move to other machines. Then, you upgrade the segregated cluster while your users use the machines and databases running your old stuff. Bring this new cluster online and let users start using it. Repeat with the other clusters, and then resume db replication. It's tricky, but possible, especially if your users will be okay if some of them run the old version while others get the new version -- /and/ if they don't need to share data immediately (i.e. the time delay before all databases are sync'd up is tolerable). Then again, there's always the weekend, when most folks aren't working, anyway ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkjSfrUACgkQ9CaO5/Lv0PA90gCfY6mqVVdb80FOjoyz95Dlb5Lt +2gAni0v8tzPtG5IoDSAC37D0V4FiPx+ =lTF9 -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Server Maintenance Across Timezones (global)
There's a lot of good suggestions here, maybe you could also justify maintaining a separate instance for the American customers. That would at least allow at a minimum to roll out changes specific for them, conform to their maintenance time :P. Yes I do realize it would be a replication of code in terms of releases but it is something to think about. On Thu, Sep 18, 2008 at 5:05 AM, Bill Davidson [EMAIL PROTECTED] wrote: My company's main webapp is used around the world (Europe, North America, Australia, etc.). We're using Tomcat as our app server and Oracle (10g) for our database. When we want to do an upgrade, that usually involves DDL changes to the database as well as corresponding changes to the webapp which means we have to make our users log out so we can shut down the app, update the DDL and restart the updated webapp. The changes are interdependent. It's all or nothing. This was not a big problem when we were just doing business in the U.S. We'd do upgrades late at night when nobody (or hardly anyone) was using the system. The problem now is that late at night here is middle of the day in other places and downtime in the middle of the day is a real problem. Our customers use our app to run parts of their business so downtime in the middle of the day is very very bad. They understandably don't like telling their customers: I'd like to help you but I need to wait for the Americans to upgrade their systems. I'm not sure how to deal with this. I've been trying to think of a way to use multiple servers and multiple databases but that seems like a synchronization nightmare. Losing data consistency is not an option. I'm sure that plenty of others on this list have had to deal with this problem. Any suggestions? How have others dealt with it? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Server Maintenance Across Timezones (global)
Hi Bill Yes, we have the same problem and I've been working on ways to improve the situation. Unfortunately, I don't think there is an easy or simple solution - a lot will depend upon your application. As far as the database side of it goes it seems to me that much of it is a question of making the 'live-update' a design requirement for any upgrades. You have to make it possible for the changes to the database to 'co-exist' and then update the live database independently of the application. When the new tables are ready, deploy your new application and (hopefully) go home smiling. We tend to create DDL changes as SQL scripts - test them out on the development systems and then run them on the live site. Some examples make it easier to see what I mean. A 'simple' change might be just adding an extra field with a static default. Obviously easy - you just run the script, it adds the extra field, the old app. doesn't have any knowledge of it. You make sure the script correctly initializes the field. When the update app runs its there. A more complex change is where a new field is required that may need to be derived from existing data. Since this data may be changing on the fly, you have to ensure consistency. The best way I've found for this is to create an 'update' transaction in your ORM code (or whatever you are using) that detects that the DDL state has changed and then runs an ACID initialization of the new field. Once again, the testing of this update is a key part of your release testing strategy. As for tomcat itself its rather going to depend whether you run clustered or not. One method that I've used on unclustered systems is to configure the new system on a server instance on a different IP address on a system fronted by httpd, set up some redirects in httpd and just, as it were, 'hup' from the old instance to the new instance. This requires a DNS change, but any 'old' requests to the old server are redirected to the new server. After about 48 hours you should be able to shut down the old server. However, we were able to catch an 'idle' period - you may not be able to do that and you'd have to ensure that any sessions active on the old server were correctly propagated to the new server. I'd be interested to hear from people who have clustered solutions. Once again I suspect there may be problems with trying to sustain sessions across the upgrades. Regards Alan Chaney Bill Davidson wrote: My company's main webapp is used around the world (Europe, North America, Australia, etc.). We're using Tomcat as our app server and Oracle (10g) for our database. When we want to do an upgrade, that usually involves DDL changes to the database as well as corresponding changes to the webapp which means we have to make our users log out so we can shut down the app, update the DDL and restart the updated webapp. The changes are interdependent. It's all or nothing. This was not a big problem when we were just doing business in the U.S. We'd do upgrades late at night when nobody (or hardly anyone) was using the system. The problem now is that late at night here is middle of the day in other places and downtime in the middle of the day is a real problem. Our customers use our app to run parts of their business so downtime in the middle of the day is very very bad. They understandably don't like telling their customers: I'd like to help you but I need to wait for the Americans to upgrade their systems. I'm not sure how to deal with this. I've been trying to think of a way to use multiple servers and multiple databases but that seems like a synchronization nightmare. Losing data consistency is not an option. I'm sure that plenty of others on this list have had to deal with this problem. Any suggestions? How have others dealt with it? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] !DSPAM:48d172ca19921381456296! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Server Maintenance Across Timezones (global)
I think the ideal approach here (potentially) is segregating your customer base. Here's an idea directly from how Salesforce.com does it. Segregate, geographically, your customer base's target infrastructure. The way they do this is by tying their customers to a specific cluster of their cloud, and then everything that customer does in the application is tied back to that cluster. The layer of redundancy (on top of being a cluster of course), is having a hot failover infrastructure that is synched with the production infrastructure at whatever feasible cycle works for the amount of data. By doing this, they can then schedule maintenance windows geographically, to ensure that impact is low no matter where the customer is. In your case, this would likely require some effort in architecting the data storage part of things to be partition-able to some extent, but this would really be maintaining what would be the effect of multiple datacenters/clusters/clouds. The only alternative I can personally offer is ensuring that the intended webapp upgrade is backwards compatible, and that the intended database upgrade is backwards/forwards compatible, so you can roll them separately (which would probably be more of a challenge than geo-separate environments). Paul McGurn -Original Message- From: Bill Davidson [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 17, 2008 5:06 PM To: Tomcat Users List Subject: Server Maintenance Across Timezones (global) My company's main webapp is used around the world (Europe, North America, Australia, etc.). We're using Tomcat as our app server and Oracle (10g) for our database. When we want to do an upgrade, that usually involves DDL changes to the database as well as corresponding changes to the webapp which means we have to make our users log out so we can shut down the app, update the DDL and restart the updated webapp. The changes are interdependent. It's all or nothing. This was not a big problem when we were just doing business in the U.S. We'd do upgrades late at night when nobody (or hardly anyone) was using the system. The problem now is that late at night here is middle of the day in other places and downtime in the middle of the day is a real problem. Our customers use our app to run parts of their business so downtime in the middle of the day is very very bad. They understandably don't like telling their customers: I'd like to help you but I need to wait for the Americans to upgrade their systems. I'm not sure how to deal with this. I've been trying to think of a way to use multiple servers and multiple databases but that seems like a synchronization nightmare. Losing data consistency is not an option. I'm sure that plenty of others on this list have had to deal with this problem. Any suggestions? How have others dealt with it? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Server Maintenance Across Timezones (global)
I get around my the same kinds of problems by keeping all the layers of the web app seperate so that i can swap them out one at a time and create a near seemless upgrade. The layers in my web apps are: 1 The web interface. 2. The application logic. (this may itself be several layers in itself if the app is complicated) 3. The database access layer. 4. The database. The key in your case is in the database access layer. The database access layer should be programmed to read either database format and write to the new one (filling in defaults/placeholders where necessary). All you need to do then is setup the structure for the new database on a new server (can use the server for testing while your at it). Drop in the new database backend. Do the rest of the data migration in the background safe in the knowledge that your webapp is still able to do whatever it needs with the old data and is already sending data to your new database aswell. The frontend can then be changed whenever (if at all). The key to any seemless upgrade is layers in the same way that redundency provides layers for server downtime. Hope what i said is useful. John5342 2008/9/17 Bill Davidson [EMAIL PROTECTED] My company's main webapp is used around the world (Europe, North America, Australia, etc.). We're using Tomcat as our app server and Oracle (10g) for our database. When we want to do an upgrade, that usually involves DDL changes to the database as well as corresponding changes to the webapp which means we have to make our users log out so we can shut down the app, update the DDL and restart the updated webapp. The changes are interdependent. It's all or nothing. This was not a big problem when we were just doing business in the U.S. We'd do upgrades late at night when nobody (or hardly anyone) was using the system. The problem now is that late at night here is middle of the day in other places and downtime in the middle of the day is a real problem. Our customers use our app to run parts of their business so downtime in the middle of the day is very very bad. They understandably don't like telling their customers: I'd like to help you but I need to wait for the Americans to upgrade their systems. I'm not sure how to deal with this. I've been trying to think of a way to use multiple servers and multiple databases but that seems like a synchronization nightmare. Losing data consistency is not an option. I'm sure that plenty of others on this list have had to deal with this problem. Any suggestions? How have others dealt with it? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Server Maintenance Across Timezones (global)
Paul McGurn wrote: Segregate, geographically, your customer base's target infrastructure. The way they do this is by tying their customers to a specific cluster of their cloud, and then everything that customer does in the application is tied back to that cluster. The layer of redundancy (on top of being a cluster of course), is having a hot failover infrastructure that is synched with the production infrastructure at whatever feasible cycle works for the amount of data. By doing this, they can then schedule maintenance windows geographically, to ensure that impact is low no matter where the customer is. Indeed. In your case, this would likely require some effort in architecting the data storage part of things to be partition-able to some extent, but this would really be maintaining what would be the effect of multiple datacenters/clusters/clouds. The main difficulty with this is consistency. Many parts of our data are tagged with dynamically generated id's (order id's for example) that are printed out and referenced by our customers, and even their customers. Running on multiple databases allows for the possibility (at this point certainty) of generating duplicate ids across different regions. This could result in a lot of confusion, particularly for support calls. We may need to learn to live with that but I still am not crazy about it. This may be the only reasonable way to do this without completely re-architecting our app (which is not really doable any time soon). - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Server Maintenance Across Timezones (global)
Easy solution to that one: Make order ID's GUIDs (globally unique identifiers). Most platforms allow for this, so I presume that Java is one of them. This could also be achieved by adding another element to the order ID generation as well. For instance, if your order ID was comprised of a few letters and the data, you could multiply the numeric part by a random number, or something to that effect. The only hole in that though is pre-existing data, but realistically, if you don't already have duplicates, it should not be a problem. The problem would arise if you have any dependency in the application on how the ID if formatted. If that's your only concern, you're doing good so far! Paul McGurn -Original Message- From: Bill Davidson [mailto:[EMAIL PROTECTED] Sent: Wednesday, September 17, 2008 6:45 PM To: Tomcat Users List Subject: Re: Server Maintenance Across Timezones (global) Paul McGurn wrote: Segregate, geographically, your customer base's target infrastructure. The way they do this is by tying their customers to a specific cluster of their cloud, and then everything that customer does in the application is tied back to that cluster. The layer of redundancy (on top of being a cluster of course), is having a hot failover infrastructure that is synched with the production infrastructure at whatever feasible cycle works for the amount of data. By doing this, they can then schedule maintenance windows geographically, to ensure that impact is low no matter where the customer is. Indeed. In your case, this would likely require some effort in architecting the data storage part of things to be partition-able to some extent, but this would really be maintaining what would be the effect of multiple datacenters/clusters/clouds. The main difficulty with this is consistency. Many parts of our data are tagged with dynamically generated id's (order id's for example) that are printed out and referenced by our customers, and even their customers. Running on multiple databases allows for the possibility (at this point certainty) of generating duplicate ids across different regions. This could result in a lot of confusion, particularly for support calls. We may need to learn to live with that but I still am not crazy about it. This may be the only reasonable way to do this without completely re-architecting our app (which is not really doable any time soon). - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Server Maintenance Across Timezones (global)
John5342 wrote: I get around my the same kinds of problems by keeping all the layers of the web app seperate so that i can swap them out one at a time and create a near seemless upgrade. The layers in my web apps are: 1 The web interface. 2. The application logic. (this may itself be several layers in itself if the app is complicated) 3. The database access layer. 4. The database. [...] Hope what i said is useful. I think it will be useful when we get to the point of redesigning the app from scratch. It's a bit tough to replace the data access layer of a large complex app that's been around for a long time though. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Server Maintenance Across Timezones (global)
I think it will be useful when we get to the point of redesigning the app from scratch. It's a bit tough to replace the data access layer of a large complex app that's been around for a long time though. It is indeed difficult to change a long standing app but in the long run its a better approach. Definately something for the next major overhaul. I use the same design principals on all of my web apps. Even on my own custom written servers (non http). I am currently in my 7th year of continuous uptime with on average 10 structural app changes every month and i have to do that on 64 servers. Trust me when i say the technique works and is well worth investing the time to setup. Building the initial framework for it is a pain but once its up and running then its well worth it and you will never look back. 2008/9/18 Bill Davidson [EMAIL PROTECTED] John5342 wrote: I get around my the same kinds of problems by keeping all the layers of the web app seperate so that i can swap them out one at a time and create a near seemless upgrade. The layers in my web apps are: 1 The web interface. 2. The application logic. (this may itself be several layers in itself if the app is complicated) 3. The database access layer. 4. The database. [...] Hope what i said is useful. I think it will be useful when we get to the point of redesigning the app from scratch. It's a bit tough to replace the data access layer of a large complex app that's been around for a long time though. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]