[chromium-dev] Re: [cros-dev] Proposed OOM improvements.
On Thu, Aug 5, 2010 at 9:14 AM, Will Drewry w...@chromium.org wrote: On Wed, Aug 4, 2010 at 8:11 PM, Luigi Semenzato semenz...@chromium.org wrote: I suspect there is one issue you may want to consider even before you get to the ones you mention. We've had reports of extreme slowness, and I was able to reproduce such situation in the past. The slowness (and pegged disk activity) is consistent with thrashing due to code paging. Even though we don't use swap, the kernel will still reclaim read-only executable pages since they have a backing store (the executable file). I suspect this may make the system unusable before you get into an actual OOM situation. Out of curiousity, would this still be the case if Chrome was running with rlimits? Will it still attempt to swap out read-only executable pages to keep memory use under that bar or will it just start returning malloc failures? Good question. I'll have to look into that some more. I know that cgroups will reclaim from the cgroup LRU list when it approaches the limit however, so maybe that's the way to go. One issue with setting resource limits is figuring out what to set them to. We'd have to be constantly tweaking the values whenever the system code changes (larger/smaller system memory use can come from anywhere). Seems like what you'd want to do is measure how much memory the browser and system (I'm including X and the window manager in the system) are using, and set the limits for the renderer and plugin processes to give the browser and systemsome headroom. But it would have to be dynamic, since the browser grows with more tabs, etc. My gut feeling (although I definitely could be wrong here) is that we'd end up with behavior similar to the OOM killer strategy in the long run -- renderers would die before the browser and system processes, it would just be a different cause of death. When I did some very informal testing a few months back, running chrome with 90% of the system memory and opening many, many tabs resulted in sad faces, but no thrashing. But that was unscientific and I never ended up exploring the rlimit v OOM code in the kernel (thus the question). Yes, this is my experience with limited testing as well. -Greg. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] Problem with git cl dcommit?
I'm trying to submit a chrome change, and I've done it before with 'git cl dcommit', but now when I try that, I get: -- About to commit; enter to confirm. git: 'svn' is not a git command. See 'git --help'. Did you mean one of these? fsck show Command git svn dcommit --no-rebase failed. -- Any ideas? -Greg. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] Re: Problem with git cl dcommit?
Nevermind. Seems that installing git-svn is a pre-requisite (but I didn't see it listed anywhere in the chrome pre-requisites). -Greg. On Wed, Jun 30, 2010 at 2:59 PM, Greg Spencer gspen...@chromium.org wrote: I'm trying to submit a chrome change, and I've done it before with 'git cl dcommit', but now when I try that, I get: -- About to commit; enter to confirm. git: 'svn' is not a git command. See 'git --help'. Did you mean one of these? fsck show Command git svn dcommit --no-rebase failed. -- Any ideas? -Greg. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] If 'git cl push' is throwing an exception for you, do this.
Recently a bug was introduced that broke git-cl so that it would throw an exception when trying to submit. I've fixed that bug, but it may not be auto-updating on your machine (I've seen a couple of cases of it not updating immediately -- no idea why). If this happens to you, you can go to your depot_tools directory, delete the directory git-cl-repo and then run ./git-cl with no arguments. This will re-populate your git-cl-repo directory from the head, which has the fix. -Greg. -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
[chromium-dev] Re: FYI: Upcoming O3D integration changes.
On Thu, Jul 23, 2009 at 2:46 AM, Dean McNamee de...@chromium.org wrote: Any idea on how much this increase the size of chrome.dll? Not yet - I'll let you know as soon as we're ready (some symbols are still not being referenced, so the count probably isn't accurate yet). -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] FYI: Upcoming O3D integration changes.
Hello Chromium Devs, The O3D team is working on getting O3D integrated into the Chromium build, and we're close to being able to complete our first step towards integration: To build the O3D plugin as part of the Chromium code base, and link it into Chromium DLL. It will still be a plugin, but will be added to the internal plugins list so that its entry points are found in the Chromium DLL, instead of loading a separate plugin DLL. This is a small step, and makes little practical difference except that 1) it will be bundled with Chromium, and 2) it opens the door to start integrating more fully. To achieve this first step, we have converted the O3D plugin to build using GYP, and have migrated to using all of the third_party libraries that we have in common at the same revision levels as Chromium is using. The next thing to do is to change Chromium's DEPS file to include the few remaining necessary dependencies that O3D uses and Chromium does not. These are: A vector math library (vectormath), Nixysa: an NPAPI IDL generator, gflags (the python parts, for nixysa), and native client (for the nacl imc libraries). I'm planning to map these like this: -- src/third_party/vectormath: http://o3d.googlecode.com/svn/trunk/googleclient/third_party/vectormath@; + Var(o3d_code_rev), src/third_party/nixysa/files: http://nixysa.googlecode.com/svn/trunk/nixysa@; + Var(nixysa_rev), src/third_party/npapi/files: http://nixysa.googlecode.com/svn/trunk/third_party/npapi@; + Var(nixysa_rev), src/third_party/ply/files: http://nixysa.googlecode.com/svn/trunk/third_party/ply-3.1@; + Var(nixysa_rev), # NACL has to be in this weird directory because it looks for # googleclient two levels above it. src/third_party/native_client/googleclient/native_client: http://nativeclient.googlecode.com/svn/trunk/nacl/googleclient/native_cli...@188 , src/third_party/gflags/: http://google-gflags.googlecode.com/svn/tr...@30;, -- In addition, I'll be making the Windows build of Chromium be dependent upon building O3D as part of the build process. This is really just a heads up! announcement to prompt heated discussions about the ensuing calamities that this will cause, so please feel free to comment. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Pixel layout tests and checksums
How about just running the pixel comparison only when the checksums don't match? Still not ideal, of course. -Greg. On Mon, Jun 22, 2009 at 11:30 AM, Ojan Vafai o...@chromium.org wrote: This isn't the best, but it would be easy to add a flag to run-webkit-tests that told it to always do the pixel comparison even if the checksums matched. Ojan On Mon, Jun 22, 2009 at 10:55 AM, Evan Martin e...@chromium.org wrote: Just so I'm not all negative, my suggestions after consulting with Tony: 1) Make Linux behavior match Windows, ignoring the recommendation in the comments below. 2) Rebaseline everything on Linux. :( 3) Now converting from a PNG file to expected output is easy on all three platforms: convert input.png rgba:- | swizzle_rgba_to_bgra | md5sum (Not certain if Mac uses BGRA images, though.) On Mon, Jun 22, 2009 at 10:24 AM, Evan Martine...@chromium.org wrote: Since I'm waiting for a build I sat down to implement this. But our image checksums are not checksums of the image files :(, but rather checksums of the pixels stored in the image. And Tony points out that our image checksumming is completely insane: = // Fix the alpha. The expected PNGs on Mac have an alpha channel, so we want // to keep it. On Windows, the alpha channel is wrong since text/form control // drawing may have erased it in a few places. So on Windows we force it to // opaque and also don't write the alpha channel for the reference. Linux // doesn't have the wrong alpha like Windows, but we ignore it anyway. #if defined(OS_WIN) bool discard_transparency = true; device-makeOpaque(0, 0, src_bmp.width(), src_bmp.height()); #elif defined(OS_LINUX) bool discard_transparency = true; #elif defined(OS_MACOSX) bool discard_transparency = false; #endif // Compute MD5 sum. We should have done this before calling // device-makeOpaque on Windows. Because we do it after the call, there are // some images that are the pixel identical on windows and other platforms // but have different MD5 sums. At this point, rebaselining all the windows // tests is too much of a pain, so we just check in different baselines. To be more clear, here's a table of the platforms and their behaviors. O=opaque, T=transparent. (Sorry for my ghetto proportionally-spaced table here.) Win Mac Lin cksumO T T pngO T O I conclude that on Linux, you cannot go from the PNG file back to the checksum in the presence of alpha. Just for fun I played around a bit with commands like: convert path/to/pngfile rgba:- | md5sum and wasn't able to repro the checksums I'm seeing. It looks ok from convert path/to/pngfile rgba:- | xxd -g4 (the RGBA-BGRA problem doesn't apply for this black and while png file...). In summary: tears. On Mon, Jun 22, 2009 at 3:20 AM, Dean McNameede...@chromium.org wrote: Last week I updated our DEPS to pull in a newer version of Skia. I was stumped at a few cases where the checked in PNG looked completely wrong, but yet it was passing on the buildbots. There was no way that image could have been the output. It just dawned on me today, but I haven't verified it. I can dig up my commit to verify it, but I'd say 99% sure this was the case. If the checksum is valid, we don't even go to the PNG. Therefor I believe we have a bunch of layout tests where the checked in PNG is completely wrong, but the checksum is right. I don't have the time right now, but it would be great if someone could write a script and clean this up. Thanks -- dean --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: 2 questions about SVN
Try using Tortise SVN, it has built-in visual diff and explorer shell integration (and it's free). http://tortoisesvn.tigris.org You might even be able to set the diff program it uses, I'm not sure (I'm on my Mac right now, but I use Tortise on my Windows machine). Note that this is only the SVN part of things -- you'll still have to use gclient to do some operations on the whole tree (like sync the dependencies). -Greg. On Fri, May 29, 2009 at 6:29 PM, nakro yoav.zilberb...@gmail.com wrote: thanx evan, just to be clear i use *windows* (hence the windiff thing) and i have svn 1.6.1 (the normal env i hope, i did not modify it) 1- svn diff `find . -name *.cc` returns: svn.exe: invalid option character: n 2- 'patch' is not recognized as an internal or external command operable program or batch file. and indeed there is no patch utility i haven't used unix in years, but as it seems your solution is unix based, and thoughts are welcome --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: reminder: don't use CHECK()
On Wed, May 27, 2009 at 11:56 AM, Amanda Walker ama...@chromium.org wrote: I'd be much happier if DCHECK exited immediately rather than relying on the code to crash, or we insisted that code using DCHECK *always* attempt to recover. Yeah, it seems that relying on the code to crash by itself after a DCHECK is unreliable. It might crash, or it might just do some truly random things (wiping out preferences or URLs or any other persistent data) that the user might not notice until it was too late, not to mention security holes (well, I guess Amanda already mentioned them :). I think that after a DCHECK you should either recover completely (which is probably not possible unless the DCHECK could be replaced by some automatic recovery code anyhow) or crash so we get some info as to why it happened. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Wed, May 13, 2009 at 7:24 PM, Brett Wilson bre...@chromium.org wrote: You can't actually canonicalize a filename on Windows, so I think it's dangerous to write a component that claims to do it. You can do it under controlled conditions, and especially if the file exists on the disk already and is accessible. For instance, if you don't try to handle (non-deterministic) 8.3 names of files that don't exist yet/anymore and NTFS mount points, I think you can fairly safely apply the regular rules to canonicalize paths (and even if you applied the rules to those, most of the time they would still work). I would make sure that the class only claims to canonicalize paths that it really knows it can do, of course. Look, I know there are tough problems here, but why not TRY to solve them as well as possible. FilePath is fine for simple manipulations, and is a good, lightweight container if you're not planning on doing anything complex with the file names. If you actually need to do more interesting things with them, like display the names, convert to relative paths, compare them for equality or pass them off to a third party in a particular encoding, it's not sufficient. I could write a half-assed implementation that kinda works if you don't throw anything wonky at it. I've got that now. I want something more bulletproof. It can't be perfect because file paths are non-deterministic on all three systems in not so obvious ways, but why should everyone who needs more than FilePath have to climb that learning curve? And we can only give out information that is as good as we get from the OS -- if the OS isn't able to present a filesystem that makes sense, we can only provide the best gibberish we can get our hands on. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: New Jank label in issue tracker
Got it. Thanks. -Greg. On Thu, May 14, 2009 at 4:49 PM, Evan Martin e...@chromium.org wrote: http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/jankometer.h?view=markup On Thu, May 14, 2009 at 4:44 PM, Greg Spencer gspen...@google.com wrote: So, what is your nuanced definition of Jank? All it means to me is bad. Aren't all bugs bad? -Greg. On Thu, May 14, 2009 at 2:44 PM, Linus Upson li...@google.com wrote: I think this is important. I've been noticing the jank level rising as well. Linus On Wed, May 13, 2009 at 5:18 PM, Ben Goodger (Google) b...@chromium.org wrote: Though I've upgraded my desktop PC to fancy hardware, I still use a Core 2 Duo 2.4GHz laptop with 2GB RAM. This is still a pretty typical configuration for even high end laptops today. Sad to say under heavy load (several windows, many tabs in each), the amount of jank I'm experiencing is pretty high. To help identify and triage these issues, I've added a Jank label in the bug system. I feel like we should focus on reducing these issues in the next release cycle - this is an area where we have historically had a good performance, we should not lose it. It's hard to write tests for jankiness because time and entropy are typically the key inputs more than any specific set of pages. But if you experience jank in your own use of Chrome I encourage you to try and identify causes and file bugs with this new label. http://code.google.com/p/chromium/issues/list?can=2q=label%3AJank -Ben --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
(ping) So, I had another idea. How about a separate file path manipulation class that has a well defined character encoding, so that we can do filename manipulations like with FilePath (and a few more). It could convert from a FilePath if given an encoding, and convert back to a FilePath with the platform's default encoding (using LC_*/LANG on Linux, falling back to ASCII), or a given encoding. It could touch the filesystem so that it could know what ecoding methods and manipulations were valid for the platform/drive combination. Since it seems like this is not really something that Chromium needs or wants right now (and it doesn't belong in base anyhow because of needing to touch the filesystem), I think I'll work on this for O3D, and later you can see if you want to use it for Chromium. -Greg. On Wed, Apr 29, 2009 at 3:58 PM, Greg Spencer gspen...@google.com wrote: On Wed, Apr 29, 2009 at 12:22 PM, Mark Mentovai m...@chromium.org wrote: I understand your problem. You're saying I have user-supplied data that I want to build a filename from, and I have this pathname that I want to display back to the user. I agree that it would be good to have a way to handle these cases in base. I don't know if FilePath proper is the right place to do it. If we do it in FilePath, it still won't really be right. OK, so it sounds like you're telling me not to use FilePath to represent file paths from a disk for my purposes because they can't ever be converted reliably to a particular encoding on Linux (which is a requirement for me, because of the third party libraries that require a particular encoding). That's fine, but what do I do instead? Roll my own FilePath clone that has some encoding assumptions? I can do that, but it has the same issues as the ones you're worried about with FilePath, so it seems better to solve the issue in one place rather than have two versions that are both insufficient. Man, it would be better if FilePath could reliably know its encoding! (I realize that Linux makes this impossible, it just seems like it would be better that way. :-) Since Linux is the only platform where the encoding is unclear, what if we did the best we could on Linux: When constructing a FilePath from a char* string on Linux: - Test the input string for values 127 to determine if it's really just ASCII (and if so, we're out of the woods). - Then check LANG, LC_CTYPE, LC_ALL (through appropriate Linux APIs) for an encoding that we can support, and note the encoding for later if we are requested to do a conversion. - If we run into an invalid sequence during a conversion, or an encoding we can't convert from, then use a CHECK to crash. This should work on most filenames, in almost all situations -- I'll bet most filenames are ASCII, even on foreign systems, and the ones that aren't ASCII have set LANG to something in /etc/profile, so all filenames created by any app running on that machine should match that encoding. Where they don't do that correctly, they're already getting garbage (and should expect garbage) from any application they use, not just Chrome, since there is no way *any *app can decode a path with multiple encodings in it, or where the encoding is different than LANG (or LC_*) says it is. Chrome already crashes like this when it encounters situations where it's just impossible to know what's right, so it's consistent with Chrome's behavior in other areas. it should be the caller's responsibility to only deal with user-created names with this interface. What do you mean here? Isn't that the case now with FilePath? (It's the file_util routines that actually read the filesystem and make FilePaths out of them, afterall). As for your suggestion to only deal with path components, how would you propose to parse user-supplied paths into one of these? 2) I'd like to make it possible to instantiate a POSIX FilePath object on Windows and a Windows FilePath on POSIX platforms. This is because some libraries (e.g. the zip library, or tar files), use POSIX semantics for their paths even on Windows (I haven't seen a use case for Windows paths on POSIX yet, actually). This would make it possible to use the nice API that FilePath has to manipulate paths appropriately for these other libraries. This could be easily accomplished by having POSIX and Windows versions of FilePath, and then typedef'ing FilePath differently on different platforms to one of these versions. Sounds pretty Pythonic. FilePath already sort of has some support for this - it does a bunch of things based on feature macros, mostly so that as I was writing it, I could test the Windows semantics without having to (shudder) resort to running on Windows. These could probably be adapted to do what you're asking. Cool. 3) It would be helpful to have real path normalization for each of the platforms (although I know what a testing nightmare
[chromium-dev] Re: Changes to FilePath?
On Wed, May 13, 2009 at 1:03 PM, Mark Mentovai m...@chromium.org wrote: If you've got to take an arbitrary FilePath and convert it for display to the user, or take an arbitrary string in a known encoding and re-encode it for the filesystem, then we don't have anything in FilePath for this. I believe that if we do add something, it should strictly operate only on single pathname components at a time, and not entire pathnames. We could add it to FilePath or we could add it somewhere else, because it is sort of distinct from what FilePath is really supposed to be, which is just a container for ferrying around native paths. OK, I can see the allure of dealing in terms of lists of encoded strings so that you can encode them separately. For my purposes, I need to get a string encoded as UTF16 (on Windows) or UTF8 (on other platforms) that represents a filename so that I can pass it to third party APIs, so it has to include the path separators. But that can be done as a join operation when I get the string out. It's also a specification and implementation nightmare. Everyone has a different idea of what normalization means. What's your idea? Yes, I know it's a nightmare all around, but I think it would be useful to have something that addresses this. My idea would be the same as Python's os.path.normpath, mainly because it's a well-tested, seasoned example with test cases. Windows also has a routine for this (PathCanonicalize) that could be used (but I know it doesn't work for UNC paths). Why would it be useful? Do you want to compare paths for equality? Yes, for instance to be able to place them into a map or set and be sure I only have one entry for a particular file. And I want to be able to do absolute to relative path conversions (as far as possible, anyhow). And yes, I know that those are *really hard* to do properly, which argues even more for implementing one in a common library so that individual developers don't roll their own all the time, thinking that it is easy (and consequently producing buggy implementations). Then we should have an API that compares paths for equality. It would have to hit the disk to do so. You might need general-purpose canonization to implement that on some systems. Great, you need to hit the disk to do that too. It's fine if you want these things, but we can't put them into FilePath. It's important that FilePath remain lightweight and not make any system calls, because system calls can block and FilePath is just a data carrier. Which is why I proposed in my last message not putting them into FilePath, since I can see that it is not your intention that it support anything that hits the filesystem (and I can see why you would want that). os.path.normpath is known to be buggy. It might be well-tested and seasoned, but only within the confines of its known limitations. Watch this. [...] Yes, I'm aware that you can create situations (especially with symbolic links) where the same path conversions will succeed or fail depending on the filesystem contents. This is why the class would have to have access to the filesystem. Again, it sounds like what you really want is a pathname comparator that hits the disk. You really can't do this stuff correctly on most systems without talking to the filesystem. You can't even do general-purpose canonization without talking to the filesystem. Yep. Totally agreed. (and normcase is probably not the behavior I'm looking for, you're right). Let me make clear: I'm not trying to shoot down the idea of needing to be able to compare paths or even necessarily canonize them. I'm arguing primarily against doing it in FilePath, but I'm also also trying to illustrate that doing proper comparisons and canonization is harder than it seems, that even seasoned and well-tested APIs are limited in ways that developers don't necessarily expect, and that the semantics and expectations need to be well-defined. Very well illustrated, and I assure you that I'm well aware that it's a bitch to do right. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Wed, May 13, 2009 at 2:05 PM, Darin Fisher da...@chromium.org wrote: That conversion is not defined. If you are on Linux, the contents of the file path is just an array of bytes. It might be UTF-8, in which case you can convert to UTF-16. However, it may also be some crazy encoding or it may not match any encoding. This OS does not require it to match an encoding. When we need to convert a FilePath to Unicode, we use the SysWideToNativeMB and SysNativeMBToWide functions from base. This works by inspecting what the system thinks the current multi-byte encoding is. On Mac that is UTF-8. On Linux, it depends on the value of $LANG. Each time we do such a conversion, we are introducing a potential bug in the product (on Linux at least), so we try hard to avoid them. Yes, I know that this is how it works (see earlier messages in this thread), but can you tell me if there are any Linux apps that manage to do this correctly (e.g. without having this bug), and how they do it? I can't see how any Linux app can do any better than looking at LANG and LC_CHAR and hoping that they're set correctly. Certainly there's no way to decode a pathname that includes multiple encodings, and I have no idea what happens with NFS mounts between machines with different settings. I'm just saying why not just do as well as can be done by the best app out there, and punt after that? -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Wed, May 13, 2009 at 4:35 PM, Darin Fisher da...@chromium.org wrote: The solution is to not convert to UTF-16 unless you are trying to generate a string to display to the user. Then you should use the LANG information to determine how best to render the text for display to the user. Yeah, that would be nice, and I agree, but the reason I need it is that some third party APIs (probably wrongly) take UTF16 to represent an input file in their API. So in order for the third party API to load the file properly, I need a UTF16 version of the file path. Also, in all of the O3D code, we assume that strings are encoded in UTF8 (which is fine and correct for any string except for filenames on Linux), so any string that might come from the user would come in as UTF8, and I'd have to translate it into a FilePath (somehow). I know this doesn't really help. I think it is reasonable to have a utility somewhere to perform a conversion to UTF-16 (or UTF-8), but it should come with a stern warning, and I kind of prefer it not being a method on FilePath since I would prefer people not be tempted to overuse it. Yeah, I think we've beat that to death: it won't be in FilePath. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Wed, May 13, 2009 at 4:07 PM, Brett Wilson bre...@chromium.org wrote: On Wed, May 13, 2009 at 3:51 PM, Amanda Walker ama...@chromium.org wrote: Perhaps what we need is a companion to FilePath. For example: FilePath: much as it is now, lightweight, alternative to string manipulation. FileReference: heavierweight, can talk to the file system and have carnal knowledge of platform specifics for things like resolving / canonicalizing pathnames, determining whether or not they refer to the same files, generating C strings that can be passed to 3rd party libraries, etc. I think this is very dangerous. I think Greg should not be talking to the filesystem when inserting filenames into a set. We don't allow filesystem access from the UI thread of Chrome, and I think other parts of our system should also not do filesystem access on their critical threads, especially if they want to be more part of Chrome in the future. Well, so the use I have for this in O3D at the moment is in our importer, which currently is a separate command-line tool that reads Collada files and writes out our wire format for geometry. So it isn't meant to be occuring in a UI thread, but I could see times when it might be useful to know for sure if two files reference the same file in the UI thread (dragging and dropping a file onto a drop zone, for instance). I do need to know if I have the same file more than once in a set because the COLLADA file might reference the same texture multiple times, or (more dangerous) it might reference a file that is one file on Windows, but (incorrectly) maps to two different files in the (Unix-path-format) .tgz files. To detect that, I need canonicalization. I also need to convert paths in the Collada file to relative paths in our tgz files. In order to do that, I need to be able to normalize the path to the Collada file so I can normalize the paths to the referenced texture files and strip off common base directories. I'd really like to avoid the filesystem access too -- it's a real pain in the ass to do, which is why it hasn't been done yet. Currently, the user has to tell me the string to strip off of the pathnames to make them relative, and if files collide or split, then the output is just 2x bigger, or just doesn't work. I'd like to fix those things, but to do it right, I need a better set of tools, and it seemed to me that if I was needing these tools, then someone else could use them too. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: depot_tools is moving!
After checking out the latest from the new location, when I run 'gclient' now, I just get the following message (on Vista 64-bit): The system cannot find the batch label specified - SVN_FORCE -Greg. On Thu, May 7, 2009 at 8:56 AM, Marc-Antoine Ruel mar...@chromium.orgwrote: Please visit http://subversion.tigris.org to download the latest subversion client before continuing. You can also get the prebacked version used at http://src.chromium.org/svn/trunk/tools/third_party/ The thing that is missing now is to stop the message from happening you need to create depot_tools\svn.bat. Feel free to make it call anything. The default one is in depot_tools\bootstrap\win\svn.new.bat. Just copy it over to depot_tools\svn.bat. I'll try to make more resilient in the coming days. M-A On Thu, May 7, 2009 at 7:58 AM, avcoder ffm...@gmail.com wrote: I fail to update I get the following message: E:\depot_toolsgclient help Installing subversion ... [-] ADODB.Stream 800a01ad: Cannot save file (Automation server can't create object) ... Failed to checkout svn automatically. Please visit http://subversion.tigris.org to download the latest subversion client before continuing. You can also get the prebacked version used at http://src.chromium.org/svn/trunk/tools/third_party/ On May 2, 4:34 am, Marc-Antoine Ruel mar...@chromium.org wrote: gcl, gclient and friends are moving fromhttp:// src.chromium.org/svn/trunk/depot_tools/tohttp://src.chromium.org/svn/trunk/tools/depot_tools/ To help you with the switch, there is now a little script to switch automatically. Just run ** *convert_depot_tools* to convert the depot_tools to the new checkout. Warning: the output of this tool isn't nice. If anything fails, just checkout manually: *svn co **http://src.chromium.org/svn/**trunk/tools/depot_tools* http://src.chromium.org/svn/trunk/tools/depot_tools The end result is the same. The far biggest advantage is that there is only one place for all the scripts (no more platform specific) and you can now send patches directly from your depot_tools; e.g. no need to do a separate checkout and run scons anymore. The old depot_tools is scheduled to be removed on May 12, 2009 and you will be upgraded *automatically* next time you run gclient after that date. Windows-only side-effects: - It won't install svn client nor python if they are found in %PATH%. - If you used svn lately, the old depot_tools version was upgraded to 1.6. If you have svn 1.5 in your %PATH%, you may have trouble working with your checkout. Just removing your old client from the path and run gclient help again. M-A --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Wed, Apr 29, 2009 at 12:22 PM, Mark Mentovai m...@chromium.org wrote: I understand your problem. You're saying I have user-supplied data that I want to build a filename from, and I have this pathname that I want to display back to the user. I agree that it would be good to have a way to handle these cases in base. I don't know if FilePath proper is the right place to do it. If we do it in FilePath, it still won't really be right. OK, so it sounds like you're telling me not to use FilePath to represent file paths from a disk for my purposes because they can't ever be converted reliably to a particular encoding on Linux (which is a requirement for me, because of the third party libraries that require a particular encoding). That's fine, but what do I do instead? Roll my own FilePath clone that has some encoding assumptions? I can do that, but it has the same issues as the ones you're worried about with FilePath, so it seems better to solve the issue in one place rather than have two versions that are both insufficient. Man, it would be better if FilePath could reliably know its encoding! (I realize that Linux makes this impossible, it just seems like it would be better that way. :-) Since Linux is the only platform where the encoding is unclear, what if we did the best we could on Linux: When constructing a FilePath from a char* string on Linux: - Test the input string for values 127 to determine if it's really just ASCII (and if so, we're out of the woods). - Then check LANG, LC_CTYPE, LC_ALL (through appropriate Linux APIs) for an encoding that we can support, and note the encoding for later if we are requested to do a conversion. - If we run into an invalid sequence during a conversion, or an encoding we can't convert from, then use a CHECK to crash. This should work on most filenames, in almost all situations -- I'll bet most filenames are ASCII, even on foreign systems, and the ones that aren't ASCII have set LANG to something in /etc/profile, so all filenames created by any app running on that machine should match that encoding. Where they don't do that correctly, they're already getting garbage (and should expect garbage) from any application they use, not just Chrome, since there is no way *any *app can decode a path with multiple encodings in it, or where the encoding is different than LANG (or LC_*) says it is. Chrome already crashes like this when it encounters situations where it's just impossible to know what's right, so it's consistent with Chrome's behavior in other areas. it should be the caller's responsibility to only deal with user-created names with this interface. What do you mean here? Isn't that the case now with FilePath? (It's the file_util routines that actually read the filesystem and make FilePaths out of them, afterall). As for your suggestion to only deal with path components, how would you propose to parse user-supplied paths into one of these? 2) I'd like to make it possible to instantiate a POSIX FilePath object on Windows and a Windows FilePath on POSIX platforms. This is because some libraries (e.g. the zip library, or tar files), use POSIX semantics for their paths even on Windows (I haven't seen a use case for Windows paths on POSIX yet, actually). This would make it possible to use the nice API that FilePath has to manipulate paths appropriately for these other libraries. This could be easily accomplished by having POSIX and Windows versions of FilePath, and then typedef'ing FilePath differently on different platforms to one of these versions. Sounds pretty Pythonic. FilePath already sort of has some support for this - it does a bunch of things based on feature macros, mostly so that as I was writing it, I could test the Windows semantics without having to (shudder) resort to running on Windows. These could probably be adapted to do what you're asking. Cool. 3) It would be helpful to have real path normalization for each of the platforms (although I know what a testing nightmare that can be). I might try and tackle this if people think it would be beneficial. It's also a specification and implementation nightmare. Everyone has a different idea of what normalization means. What's your idea? Yes, I know it's a nightmare all around, but I think it would be useful to have something that addresses this. My idea would be the same as Python's os.path.normpath, mainly because it's a well-tested, seasoned example with test cases. Windows also has a routine for this (PathCanonicalize) that could be used (but I know it doesn't work for UNC paths). 4) Make sure we handle case sensitivity vs case preservation correctly. It's unclear to me that FilePath does this correctly on the Mac -- Mac file names are case preserving, but case insensitive, Unix filenames are both (and windows filenames are neither :-). Again with the normalization. What do you want this stuff for? What's your
[chromium-dev] Changes to FilePath?
Hi Chromium Developers, I'm working on Google's O3D (http://code.google.com/p/o3d), and we (naturally) share some of Chrome's base classes for our code, including the very useful class FilePath. However, in using FilePath in the last few months, I've seen that it needs some refinement. I'd like to augment the FilePath class with some things that would make it more generally useful -- it's very nicely set up, but it's missing a few things that make it harder to work with than it needs to be: 1) I'd like to add some explicit routines for converting to/from UTF8 and UTF16. While it's nice (and important) that FilePath uses the platform's native string, we've found that many third party libraries have made other assumptions, where they always expect UTF8 (char) or UTF16 (wchar_t) paths regardless of platform, and converting a FilePath to and from those forms is a platform-dependent exercise which should be centralized into the class (i.e. adding ToUTF8 and ToWide functions to the class, and explicit constructors that take each type). 2) I'd like to make it possible to instantiate a POSIX FilePath object on Windows and a Windows FilePath on POSIX platforms. This is because some libraries (e.g. the zip library, or tar files), use POSIX semantics for their paths even on Windows (I haven't seen a use case for Windows paths on POSIX yet, actually). This would make it possible to use the nice API that FilePath has to manipulate paths appropriately for these other libraries. This could be easily accomplished by having POSIX and Windows versions of FilePath, and then typedef'ing FilePath differently on different platforms to one of these versions. 3) It would be helpful to have real path normalization for each of the platforms (although I know what a testing nightmare that can be). I might try and tackle this if people think it would be beneficial. 4) Make sure we handle case sensitivity vs case preservation correctly. It's unclear to me that FilePath does this correctly on the Mac -- Mac file names are case preserving, but case insensitive, Unix filenames are both (and windows filenames are neither :-). So, is there any resistance to any of the above? Do you have other suggestions that I might take into account? Am I violating any design assumptions of FilePath? For #2, is speed/size enough of a concern to avoid a virtual base class (I wouldn't think so, but you never know..)? -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Tue, Apr 28, 2009 at 1:57 PM, Thomas Van Lenten thoma...@chromium.orgwrote: On Tue, Apr 28, 2009 at 4:39 PM, Greg Spencer gspen...@google.com wrote: 4) Make sure we handle case sensitivity vs case preservation correctly. It's unclear to me that FilePath does this correctly on the Mac -- Mac file names are case preserving, but case insensitive, Unix filenames are both (and windows filenames are neither :-). FYI - it's a drive format time option on the Mac, so they can be case preserving and case sensitive. Thanks for pointing that out. In fact, NTFS is actually case sensitive, where FAT32 is not (see http://support.microsoft.com/kb/100625). So we have issues there as well. The real issue would be dealing with relative paths that don't exist yet -- there would be no way to inspect the file location to find out what mode it was in. I think I would just punt and go with the widely-used defaults (the ones I mentioned above), since most apps seem to assume those limitations. An alternative would be to have an API to specify the desired mode, and default to the common case on each platform. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Tue, Apr 28, 2009 at 2:31 PM, Peter Kasting pkast...@google.com wrote: On Tue, Apr 28, 2009 at 1:39 PM, Greg Spencer gspen...@google.com wrote: 1) I'd like to add some explicit routines for converting to/from UTF8 and UTF16. While it's nice (and important) that FilePath uses the platform's native string, we've found that many third party libraries have made other assumptions, where they always expect UTF8 (char) or UTF16 (wchar_t) paths regardless of platform, and converting a FilePath to and from those forms is a platform-dependent exercise which should be centralized into the class (i.e. adding ToUTF8 and ToWide functions to the class, and explicit constructors that take each type). I'm pretty strongly against this for the same reasons as Evan. I think consumers who need to convert should be doing the conversion using their own routines (e.g. Chrome uses ones in our base/ module). So, I was unable to find the conversion utilities in base that do the conversion to/from UTF8. What are they called? If I missed them (and I looked for a while before I gave up), then maybe they need to be more prominent? What is the danger here of being lazy? Is it that developers will unwittingly do expensive conversions? If so, I would expect that a member function called ToUTF8 would be just as much of a performance warning as a helper function called FilePathToUTF8, but be a heck of a lot more convenient (since it would not require the developer to create a local variable for use as a return value from the helper, and can be used as an argument to another library's functions). I can see the argument for not having a casting constructor that isn't from the platform native form, but in that case, a factory method called CreateFromUTF8 should be a sufficient warning to the developer that it might be expensive. -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Tue, Apr 28, 2009 at 3:19 PM, Greg Spencer gspen...@google.com wrote: On Tue, Apr 28, 2009 at 3:11 PM, Erik Kay erik...@google.com wrote: The biggest problem with this change is that it's not possible to do this conversion on Linux in a safe way. And besides -- this problem isn't introduced by this change: it exists already because currently there's no safe way to convert, regardless of the API (since a consumer of a FilePath doesn't know what encoding it contains). -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Changes to FilePath?
On Tue, Apr 28, 2009 at 3:26 PM, Erik Kay erik...@chromium.org wrote: On Tue, Apr 28, 2009 at 3:19 PM, Greg Spencer gspen...@google.com wrote: But that's exactly the point. FilePath is the class that created the path to begin with. So it can know what the LC_*/LANG variables were was when it was created, and do the right conversion when you ask the FilePath to convert to UTF16. Also, if the developer calls something called FilePath::CreateFromUTF8, then it can know it was supposed to be UTF8 and remember that. If you created it yourself, that's fine. FilePaths aren't always created manually by users. They often are populated from system APIs where you can't know. See file_util* for some examples. So the problem is that if you add this API, people will mistakenly use the conversion functions when they can't be safe. I agree it sucks. I just don't know of a reasonable solution. So there's currently no right way to do the conversion, but I still think that the FilePath constructor is probably in the best position to inspect LC_ALL, etc. and do as close to the right thing as possible. I doubt most Linux developers even think about this, and so the chances that they will implement anything other than assuming that it's ASCII are slim -- this would allow us to at least implement a baseline for them. Or would that just screw things up worse? Doesn't this mean that it's possible that the path manipulation routines fail for sufficiently odd encodings? (jis or something where an encoded char might include a /?) -Greg. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
