[chromium-dev] Re: Chromium App Executables Disk Layout
How will in-place updating work on the Mac and Linux? To be frank, we haven't solved this problem on Mac. Right now we're doing an rsync to klobber on update, which is fine for pre-dogfood. E.g. our normal Mac crash rate far exceeds any possible crashes caused by version mismatching in the 3 auto-updates we have sent out internally. Although we could load all resources on startup, that ignores one critical piece. Since renderers are separate processes and are launched on-demand, we would still have the problem of old browser talking to new renderer. I suspect we'll need to have either a versioned scheme line Windows, or a complete upgrade step on initial launch. jrg --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On Sat, Mar 28, 2009 at 5:35 PM, Dan Kegel daniel.r.ke...@gmail.com wrote: On Sat, Mar 28, 2009 at 4:59 PM, Erik Kay erik...@chromium.org wrote: ...When you update in place, resources can be changed, removed, etc. So let's say that some dll hasn't been loaded yet, but then an autoupdate happens. Now you trigger loading the dll, and you crash, because this kind of forwards compatibility is never tested (nor should it be). This can happen with simple data files as well (although we don't have many of those at the moment). This is why all of the loaded code and resources are in a versioned directory (minus dictionaries, but that's a separate issue). How will in-place updating work on the Mac and Linux? For Linux, I was imagining that we'd open handles at startup to any file we'd need, and then never open any more handles after that. That way, the window during which an update could screw you would be brief (i.e. only if you started the app halfway through installation of an update could it misbehave). We looked into this more at one point and noticed that mmap doesn't make any promises when the file on disk changes. We'll probably have to use versioned directories as well to avoid problems when updating while running. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On Sun, Mar 29, 2009 at 11:56 AM, Dan Kegel d...@kegel.com wrote: How will in-place updating work on the Mac and Linux? For Linux, I was imagining that we'd open handles at startup to any file we'd need, and then never open any more handles after that. We looked into this more at one point and noticed that mmap doesn't make any promises when the file on disk changes. We'll probably have to use versioned directories as well to avoid problems when updating while running. Sounds good, but when the system updates the app, the old directory will go away. Duh. The app will still have a handle to the old file, which will remain valid even if the file is unlinked. Should work as long as we fork rather than exec. Delay loading could be problematic, though. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On Sun, Mar 29, 2009 at 1:02 PM, Thomas Van Lenten thoma...@chromium.org wrote: Most mac apps solve this by having the app exit as part of the upgrade process, this way a new copy is launched w/ the new resources. yes, but this is the problem with silent autoupdate. We don't want to force the user to stop what they're doing when an update starts. Erik On Sun, Mar 29, 2009 at 4:40 AM, John Grabowski j...@chromium.org wrote: How will in-place updating work on the Mac and Linux? To be frank, we haven't solved this problem on Mac. Right now we're doing an rsync to klobber on update, which is fine for pre-dogfood. E.g. our normal Mac crash rate far exceeds any possible crashes caused by version mismatching in the 3 auto-updates we have sent out internally. Although we could load all resources on startup, that ignores one critical piece. Since renderers are separate processes and are launched on-demand, we would still have the problem of old browser talking to new renderer. All platforms would have this problem if they don't force the apps to bounce as part of the upgrade process, no? TVL I suspect we'll need to have either a versioned scheme line Windows, or a complete upgrade step on initial launch. jrg --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
Perhaps we need to think about how to encourage such a restart, without being the traditional pop-up pain-in-the-er-um-rear. Can we find real-estate that won't tend to detract from the ui to alert the user that an update is pending? Perhaps atop a new-tab page? Perhaps an extra entry ot the top-level wrench menu? ...in a different color near About Chrome?? On windows, should we be alerting the user via the little updates are pending dialog that Windows and others are using?? Can we make the update especially painless for users, by temporarily opting them into to restore the existing tab settings, even if they don't usually do so?? Can we do other things to make it easy, and obvious to encourage a user to allow an upgrade? I don't think it is possible... but can we detect when it would be a good time to upgrade, and just help them along? I suspect that too much state would be lost (sign in, cookies, SSL session keys, etc. etc.). Maybe we can detect* that they are in a very reproducible state and that they are quiescent and then do the update (perchance a large percentage of users live in such states)?? Can we do some fancy optimization to do most of the startup of the new version before we shut down the old version, so the transition is super-fast?? It is clear to me that our wonderful work to crash less is working... and greatly extending the time-till-update after the user has the downloaded update on disk. IMO, delaying an upgrade is probably a Bad Thing (TM) for the user (which is why we push out better versions)... but interrupting a user's action is a very Bad Thing. Can we improve the compromise, and help the user? Jim On Sun, Mar 29, 2009 at 1:25 PM, Erik Kay erik...@chromium.org wrote: On Sun, Mar 29, 2009 at 1:02 PM, Thomas Van Lenten thoma...@chromium.org wrote: Most mac apps solve this by having the app exit as part of the upgrade process, this way a new copy is launched w/ the new resources. yes, but this is the problem with silent autoupdate. We don't want to force the user to stop what they're doing when an update starts. Erik On Sun, Mar 29, 2009 at 4:40 AM, John Grabowski j...@chromium.org wrote: How will in-place updating work on the Mac and Linux? To be frank, we haven't solved this problem on Mac. Right now we're doing an rsync to klobber on update, which is fine for pre-dogfood. E.g. our normal Mac crash rate far exceeds any possible crashes caused by version mismatching in the 3 auto-updates we have sent out internally. Although we could load all resources on startup, that ignores one critical piece. Since renderers are separate processes and are launched on-demand, we would still have the problem of old browser talking to new renderer. All platforms would have this problem if they don't force the apps to bounce as part of the upgrade process, no? TVL I suspect we'll need to have either a versioned scheme line Windows, or a complete upgrade step on initial launch. jrg --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
It would be nice to be able to leverage our multi-process architecture and at least start using the new binaries for any new tabs that are created. But, that's a different can of worms, I guess. On Sun, Mar 29, 2009 at 14:31, Jim Roskind j...@chromium.org wrote: Perhaps we need to think about how to encourage such a restart, without being the traditional pop-up pain-in-the-er-um-rear. Can we find real-estate that won't tend to detract from the ui to alert the user that an update is pending? Perhaps atop a new-tab page? Perhaps an extra entry ot the top-level wrench menu? ...in a different color near About Chrome?? On windows, should we be alerting the user via the little updates are pending dialog that Windows and others are using?? Can we make the update especially painless for users, by temporarily opting them into to restore the existing tab settings, even if they don't usually do so?? Can we do other things to make it easy, and obvious to encourage a user to allow an upgrade? I don't think it is possible... but can we detect when it would be a good time to upgrade, and just help them along? I suspect that too much state would be lost (sign in, cookies, SSL session keys, etc. etc.). Maybe we can detect* that they are in a very reproducible state and that they are quiescent and then do the update (perchance a large percentage of users live in such states)?? Can we do some fancy optimization to do most of the startup of the new version before we shut down the old version, so the transition is super-fast?? It is clear to me that our wonderful work to crash less is working... and greatly extending the time-till-update after the user has the downloaded update on disk. IMO, delaying an upgrade is probably a Bad Thing (TM) for the user (which is why we push out better versions)... but interrupting a user's action is a very Bad Thing. Can we improve the compromise, and help the user? Jim On Sun, Mar 29, 2009 at 1:25 PM, Erik Kay erik...@chromium.org wrote: On Sun, Mar 29, 2009 at 1:02 PM, Thomas Van Lenten thoma...@chromium.org wrote: Most mac apps solve this by having the app exit as part of the upgrade process, this way a new copy is launched w/ the new resources. yes, but this is the problem with silent autoupdate. We don't want to force the user to stop what they're doing when an update starts. Erik On Sun, Mar 29, 2009 at 4:40 AM, John Grabowski j...@chromium.org wrote: How will in-place updating work on the Mac and Linux? To be frank, we haven't solved this problem on Mac. Right now we're doing an rsync to klobber on update, which is fine for pre-dogfood. E.g. our normal Mac crash rate far exceeds any possible crashes caused by version mismatching in the 3 auto-updates we have sent out internally. Although we could load all resources on startup, that ignores one critical piece. Since renderers are separate processes and are launched on-demand, we would still have the problem of old browser talking to new renderer. All platforms would have this problem if they don't force the apps to bounce as part of the upgrade process, no? TVL I suspect we'll need to have either a versioned scheme line Windows, or a complete upgrade step on initial launch. jrg --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
This is a very delicate line to walk. We want to silently update... yet not actually be silent. We want updates to take effect immediately... yet not require an app restart. You have provided some good ideas but, in practice, it can be difficult to close on a compromise that doesn't make some subset of players very upset, or expose a new class of complexity. Chrom(ium) packaging and resource finding is different on each platform, which has implications for an update strategy. For the Mac, thomasvl is currently reworking the app structure, so it is probably best to wait for his work to complete before we make a proposal. jrg On Sun, Mar 29, 2009 at 2:31 PM, Jim Roskind j...@chromium.org wrote: Perhaps we need to think about how to encourage such a restart, without being the traditional pop-up pain-in-the-er-um-rear. Can we find real-estate that won't tend to detract from the ui to alert the user that an update is pending? Perhaps atop a new-tab page? Perhaps an extra entry ot the top-level wrench menu? ...in a different color near About Chrome?? On windows, should we be alerting the user via the little updates are pending dialog that Windows and others are using?? Can we make the update especially painless for users, by temporarily opting them into to restore the existing tab settings, even if they don't usually do so?? Can we do other things to make it easy, and obvious to encourage a user to allow an upgrade? I don't think it is possible... but can we detect when it would be a good time to upgrade, and just help them along? I suspect that too much state would be lost (sign in, cookies, SSL session keys, etc. etc.). Maybe we can detect* that they are in a very reproducible state and that they are quiescent and then do the update (perchance a large percentage of users live in such states)?? Can we do some fancy optimization to do most of the startup of the new version before we shut down the old version, so the transition is super-fast?? It is clear to me that our wonderful work to crash less is working... and greatly extending the time-till-update after the user has the downloaded update on disk. IMO, delaying an upgrade is probably a Bad Thing (TM) for the user (which is why we push out better versions)... but interrupting a user's action is a very Bad Thing. Can we improve the compromise, and help the user? Jim On Sun, Mar 29, 2009 at 1:25 PM, Erik Kay erik...@chromium.org wrote: On Sun, Mar 29, 2009 at 1:02 PM, Thomas Van Lenten thoma...@chromium.org wrote: Most mac apps solve this by having the app exit as part of the upgrade process, this way a new copy is launched w/ the new resources. yes, but this is the problem with silent autoupdate. We don't want to force the user to stop what they're doing when an update starts. Erik On Sun, Mar 29, 2009 at 4:40 AM, John Grabowski j...@chromium.org wrote: How will in-place updating work on the Mac and Linux? To be frank, we haven't solved this problem on Mac. Right now we're doing an rsync to klobber on update, which is fine for pre-dogfood. E.g. our normal Mac crash rate far exceeds any possible crashes caused by version mismatching in the 3 auto-updates we have sent out internally. Although we could load all resources on startup, that ignores one critical piece. Since renderers are separate processes and are launched on-demand, we would still have the problem of old browser talking to new renderer. All platforms would have this problem if they don't force the apps to bounce as part of the upgrade process, no? TVL I suspect we'll need to have either a versioned scheme line Windows, or a complete upgrade step on initial launch. jrg --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
I'm supportive of some UI to let the user know that there is a update pending. This should be non-model and minimally distracting. We've talked about a notification area on the NTP. It would also serve as a place that we could tell people about cool new features. We'll throw this on the queue for the UI team to mock up (if we don't have mocks already). On Sun, Mar 29, 2009 at 3:49 PM, John Grabowski j...@chromium.org wrote: This is a very delicate line to walk. We want to silently update... yet not actually be silent. We want updates to take effect immediately... yet not require an app restart. You have provided some good ideas but, in practice, it can be difficult to close on a compromise that doesn't make some subset of players very upset, or expose a new class of complexity. Chrom(ium) packaging and resource finding is different on each platform, which has implications for an update strategy. For the Mac, thomasvl is currently reworking the app structure, so it is probably best to wait for his work to complete before we make a proposal. jrg On Sun, Mar 29, 2009 at 2:31 PM, Jim Roskind j...@chromium.org wrote: Perhaps we need to think about how to encourage such a restart, without being the traditional pop-up pain-in-the-er-um-rear. Can we find real-estate that won't tend to detract from the ui to alert the user that an update is pending? Perhaps atop a new-tab page? Perhaps an extra entry ot the top-level wrench menu? ...in a different color near About Chrome?? On windows, should we be alerting the user via the little updates are pending dialog that Windows and others are using?? Can we make the update especially painless for users, by temporarily opting them into to restore the existing tab settings, even if they don't usually do so?? Can we do other things to make it easy, and obvious to encourage a user to allow an upgrade? I don't think it is possible... but can we detect when it would be a good time to upgrade, and just help them along? I suspect that too much state would be lost (sign in, cookies, SSL session keys, etc. etc.). Maybe we can detect* that they are in a very reproducible state and that they are quiescent and then do the update (perchance a large percentage of users live in such states)?? Can we do some fancy optimization to do most of the startup of the new version before we shut down the old version, so the transition is super-fast?? It is clear to me that our wonderful work to crash less is working... and greatly extending the time-till-update after the user has the downloaded update on disk. IMO, delaying an upgrade is probably a Bad Thing (TM) for the user (which is why we push out better versions)... but interrupting a user's action is a very Bad Thing. Can we improve the compromise, and help the user? Jim On Sun, Mar 29, 2009 at 1:25 PM, Erik Kay erik...@chromium.org wrote: On Sun, Mar 29, 2009 at 1:02 PM, Thomas Van Lenten thoma...@chromium.org wrote: Most mac apps solve this by having the app exit as part of the upgrade process, this way a new copy is launched w/ the new resources. yes, but this is the problem with silent autoupdate. We don't want to force the user to stop what they're doing when an update starts. Erik On Sun, Mar 29, 2009 at 4:40 AM, John Grabowski j...@chromium.org wrote: How will in-place updating work on the Mac and Linux? To be frank, we haven't solved this problem on Mac. Right now we're doing an rsync to klobber on update, which is fine for pre-dogfood. E.g. our normal Mac crash rate far exceeds any possible crashes caused by version mismatching in the 3 auto-updates we have sent out internally. Although we could load all resources on startup, that ignores one critical piece. Since renderers are separate processes and are launched on-demand, we would still have the problem of old browser talking to new renderer. All platforms would have this problem if they don't force the apps to bounce as part of the upgrade process, no? TVL I suspect we'll need to have either a versioned scheme line Windows, or a complete upgrade step on initial launch. jrg --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On Wed, Mar 25, 2009 at 6:11 PM, cpu c...@chromium.org wrote: On windows the exe/dll separation it was meant to simplify upgrade but in practice the exe changes frequently enough that we still need to support exe upgrading in place. (resending) While the exe changing issue is annoying, and we have to handle in-use file swapping, it's not the main issue with updates. When you update in place, resources can be changed, removed, etc. So let's say that some dll hasn't been loaded yet, but then an autoupdate happens. Now you trigger loading the dll, and you crash, because this kind of forwards compatibility is never tested (nor should it be). This can happen with simple data files as well (although we don't have many of those at the moment). This is why all of the loaded code and resources are in a versioned directory (minus dictionaries, but that's a separate issue). How will in-place updating work on the Mac and Linux? Erik We always wanted to have a do nothing exe that just loads the chrome dll but that never happened, reasons: 1) the sandbox 2) breakpad 3) google_update. All this things are living on the exe. We could move to a no-dll world easier than moving to a dumb exe world. On Mar 24, 9:50 pm, Craig Schlenter craig.schlen...@gmail.com wrote: On Tue, Mar 24, 2009 at 8:45 PM, Thomas Van Lententhoma...@chromium.org wrote: [snip] Linux currently builds as one executable also. But Adam proposed we create a second executable (via hardlink?) for AppArmor as a sandbox? Will a hard link work for selinux btw.? I'm under the impression that selinux is inode based so Fedora etc. would have to do something different there. As for the shared library overhead that Dean mentioned, there is some interesting work on linux that is mentioned herehttp://lwn.net/Articles/192624/... hopefully some of that has made it into the default toolchain by now. Pre-linking is enabled by default on Fedora btw. ... I'm unsure of other distributions. --Craig --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On Sat, Mar 28, 2009 at 4:59 PM, Erik Kay erik...@chromium.org wrote: ...When you update in place, resources can be changed, removed, etc. So let's say that some dll hasn't been loaded yet, but then an autoupdate happens. Now you trigger loading the dll, and you crash, because this kind of forwards compatibility is never tested (nor should it be). This can happen with simple data files as well (although we don't have many of those at the moment). This is why all of the loaded code and resources are in a versioned directory (minus dictionaries, but that's a separate issue). How will in-place updating work on the Mac and Linux? For Linux, I was imagining that we'd open handles at startup to any file we'd need, and then never open any more handles after that. That way, the window during which an update could screw you would be brief (i.e. only if you started the app halfway through installation of an update could it misbehave). - Dan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On windows the exe/dll separation it was meant to simplify upgrade but in practice the exe changes frequently enough that we still need to support exe upgrading in place. We always wanted to have a do nothing exe that just loads the chrome dll but that never happened, reasons: 1) the sandbox 2) breakpad 3) google_update. All this things are living on the exe. We could move to a no-dll world easier than moving to a dumb exe world. On Mar 24, 9:50 pm, Craig Schlenter craig.schlen...@gmail.com wrote: On Tue, Mar 24, 2009 at 8:45 PM, Thomas Van Lententhoma...@chromium.org wrote: [snip] Linux currently builds as one executable also. But Adam proposed we create a second executable (via hardlink?) for AppArmor as a sandbox? Will a hard link work for selinux btw.? I'm under the impression that selinux is inode based so Fedora etc. would have to do something different there. As for the shared library overhead that Dean mentioned, there is some interesting work on linux that is mentioned herehttp://lwn.net/Articles/192624/... hopefully some of that has made it into the default toolchain by now. Pre-linking is enabled by default on Fedora btw. ... I'm unsure of other distributions. --Craig --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
Something important would be to understand the overhead for a shared library (fpic, relocation, etc). On Tue, Mar 24, 2009 at 7:45 PM, Thomas Van Lenten thoma...@chromium.org wrote: The Windows product builds a small executable that then loads the main chromium dll, and hands off control to that. I believe this is done solely for updating reasons. All the difference processes start from that one shim. On the Mac, we are currently building as a single executable. But, this brings up some complications, we need to be able to launch the Renderers with different Cocoa initialization. This data comes out of the info.plist, so we really need different bundles on disk (the OS acts on the data before the process is even started). So what it's looking like is that we need to move to a world where we also have a small shim for Chromium that loads a main shared lib and hands off control. Then we'll have a second shim for Renderers (and maybe plugin hosts, etc.) that loads the main shared lib and hands off control. Each of these shims will have different info.plists to provide the different Cocoa configuration information. Linux currently builds as one executable also. But Adam proposed we create a second executable (via hardlink?) for AppArmor as a sandbox? Does it make sense to standardize/require a small shell and shared lib for all platforms? One advantage of this approach on all platforms is they can initialize breakpad/crash reporting to the started up in the shim, so a crash during the loading of the main shared lib would be captured. One place not standardizing this gets ugly is within the build system, where it could become very complex expressing what goes into apps vs. shared libs. Thoughts/suggestions/comments? TVL --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
Dean McNamee wrote: Something important would be to understand the overhead for a shared library (fpic, relocation, etc). On the Mac, I actually want to link the main executable with -pie so that it loads at a different address each time anyway. That implies PIC. It's still wise to understand the overhead, which is why I've been waiting until we have some perf tests running to make this change. Mark --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Chromium App Executables Disk Layout
On Tue, Mar 24, 2009 at 4:07 PM, George Djabarov geor...@google.com wrote: One additional reason to use small executable that loads chrome main dll is to prevent firewalls warnings each time chrome is updated. On the Mac, we should be able to avoid these by having a signed executable. Once the user says to trust the binary, the signature allows that to stay across updates (if the sig is the same). TVL -George On Tue, Mar 24, 2009 at 11:45 AM, Thomas Van Lenten thoma...@chromium.org wrote: The Windows product builds a small executable that then loads the main chromium dll, and hands off control to that. I believe this is done solely for updating reasons. All the difference processes start from that one shim. On the Mac, we are currently building as a single executable. But, this brings up some complications, we need to be able to launch the Renderers with different Cocoa initialization. This data comes out of the info.plist, so we really need different bundles on disk (the OS acts on the data before the process is even started). So what it's looking like is that we need to move to a world where we also have a small shim for Chromium that loads a main shared lib and hands off control. Then we'll have a second shim for Renderers (and maybe plugin hosts, etc.) that loads the main shared lib and hands off control. Each of these shims will have different info.plists to provide the different Cocoa configuration information. Linux currently builds as one executable also. But Adam proposed we create a second executable (via hardlink?) for AppArmor as a sandbox? Does it make sense to standardize/require a small shell and shared lib for all platforms? One advantage of this approach on all platforms is they can initialize breakpad/crash reporting to the started up in the shim, so a crash during the loading of the main shared lib would be captured. One place not standardizing this gets ugly is within the build system, where it could become very complex expressing what goes into apps vs. shared libs. Thoughts/suggestions/comments? TVL --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
