[chromium-dev] Re: fighting the flakiness - resource bundle issues?
On Wed, Aug 5, 2009 at 5:32 PM, Jeremy Orlow jor...@google.com wrote: Which kind soul ended up working on this? Big thanks are in order, but I wanted to let you know that at least one case slipped through the cracks: http://src.chromium.org/viewvc/chrome?view=revrevision=22556 required a clobber today. Any ideas why? It didn't require a clobber; it went green on the next run (inexplicably). PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
I'm working on gclient to pass the matching file list to the hook, or some equivalent. I discovered another issue, which I highly suspect is caused by a build issue (I'm not sure if it's the same issue). waiting_animation_frames = rb.GetBitmapNamed(IDR_THROBBER_WAITING); DCHECK(waiting_animation_frames); DCHECK(waiting_animation_frames-width() % waiting_animation_frames-height() == 0); // -- this DCHECK fails waiting_animation_frame_count = waiting_animation_frames-width() / waiting_animation_frames- height(); It seems that a wrong bitmap is being loaded. What can it mean for the build? On Mon, Jul 27, 2009 at 16:54, Paweł Hajdan Jr. phajdan...@chromium.orgwrote: I'm going to try writing the hook, but I would first ask for advice. The hook syntax takes a filename pattern and a command. So I would have to create a new command (probably in src/tools), like clobber_generated_headers or something similar. And the tool itself does not get the list of changed files, so it has to clobber all of them, and have the list of files to be clobbered hardcoded. Do you see better solutions? --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
This happens semi-frequently. It's related to the grit resources rebuilding. Something like: not remapping IDR_THROBBER_WAITING to the right place, or remapping it to the right place but not rebuilding this file, or something like that. On Tue, Jul 28, 2009 at 4:18 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: I'm working on gclient to pass the matching file list to the hook, or some equivalent. I discovered another issue, which I highly suspect is caused by a build issue (I'm not sure if it's the same issue). waiting_animation_frames = rb.GetBitmapNamed(IDR_THROBBER_WAITING); DCHECK(waiting_animation_frames); DCHECK(waiting_animation_frames-width() % waiting_animation_frames-height() == 0); // -- this DCHECK fails waiting_animation_frame_count = waiting_animation_frames-width() / waiting_animation_frames- height(); It seems that a wrong bitmap is being loaded. What can it mean for the build? On Mon, Jul 27, 2009 at 16:54, Paweł Hajdan Jr. phajdan...@chromium.org wrote: I'm going to try writing the hook, but I would first ask for advice. The hook syntax takes a filename pattern and a command. So I would have to create a new command (probably in src/tools), like clobber_generated_headers or something similar. And the tool itself does not get the list of changed files, so it has to clobber all of them, and have the list of files to be clobbered hardcoded. Do you see better solutions? --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
On Tue, Jul 28, 2009 at 4:35 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: On Tue, Jul 28, 2009 at 16:30, Evan Martin e...@chromium.org wrote: This happens semi-frequently. It's related to the grit resources rebuilding. Something like: not remapping IDR_THROBBER_WAITING to the right place, or remapping it to the right place but not rebuilding this file, or something like that. Will the workaround for the original problem (clobbering headers generated by .grd files) work also for this issue? If not, do you have any idea for a fix or workaround? Or maybe... why not set the resources projects to always-build, even just for the buildbots? lastchange works this way, so it is possible. I think it is not more or less crude than the clobbering workaround, and is simpler to implement. Increase in the build time should not be significant (on the bots; doing that on developers' machine probably isn't a good idea). I like the idea of always-building the resources project. Though note it can cause a lot of code to rebuild: it generates headers, which are included via includes via includes... --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
On Tue, Jul 28, 2009 at 4:37 PM, Evan Martin e...@chromium.org wrote: On Tue, Jul 28, 2009 at 4:35 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: On Tue, Jul 28, 2009 at 16:30, Evan Martin e...@chromium.org wrote: This happens semi-frequently. It's related to the grit resources rebuilding. Something like: not remapping IDR_THROBBER_WAITING to the right place, or remapping it to the right place but not rebuilding this file, or something like that. Will the workaround for the original problem (clobbering headers generated by .grd files) work also for this issue? If not, do you have any idea for a fix or workaround? Or maybe... why not set the resources projects to always-build, even just for the buildbots? lastchange works this way, so it is possible. I think it is not more or less crude than the clobbering workaround, and is simpler to implement. Increase in the build time should not be significant (on the bots; doing that on developers' machine probably isn't a good idea). I like the idea of always-building the resources project. Though note it can cause a lot of code to rebuild: it generates headers, which are included via includes via includes... I still vote for the original idea of clobbering the .h files if the .grd file is changed. I think it should be easy to implement with the DEPS file. The faster the incremental build is on the bots, the faster they cycle. They are already too slow, that'd be sad to make it even slower. Nicolas --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
I'm going to try writing the hook, but I would first ask for advice. The hook syntax takes a filename pattern and a command. So I would have to create a new command (probably in src/tools), like clobber_generated_headers or something similar. And the tool itself does not get the list of changed files, so it has to clobber all of them, and have the list of files to be clobbered hardcoded. Do you see better solutions? --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
Tony, just to make sure, are you certain that this is a IB only issue? I used to be convinced that it was happening with visual studio too. If this is a IB only problem, we should create a repro case and send it to them, they are usually really responsive. [For some reasons, I thought it was because it was generating the rc files and the ID in them were not static. I.e. you add a string in the middle, and it's shifting the string IDs of all the strings after the addition. Visual Studio does not support that. You can't change string ids in a RC file without having the clobber the source. The normal use case is always to take the next available ID (which is why the rc file always keep track of the next available ID at the bottom).] Nicolas On Thu, Jul 23, 2009 at 5:54 PM, Tony Chang t...@chromium.org wrote: Look at how the current gyp hook works. It looks for changes to .gyp files and only runs if a .gyp (and maybe gypi?) file has changed. You can find what header it generates by opening the grd file and parsing the XML (the XML lists the output files). You'll need to build the base directory (e.g., Debug/obj/global_intermediate/ and the Release version), I would just check for the Windows versions since we don't seem to have this problem with the mac or linux buildbots. On Thu, Jul 23, 2009 at 5:36 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: I think that this workaround may be worth it. I'm not familiar with the IncrediBuild, but I can help making the hook (and we can run it only on Windows). How do I make a hook know which grd files changed? And also know which headers it generates? Alternatively, maybe this Windows-only hook would just delete all generated headers (with a hardcoded list)? Generation seems to be cheap, and such hook seems trivial to write. So, yes, this hook is kludgey. But we are aware of its limitations, and it would eliminate one kind of build mysteries. What do you think? On Thu, Jul 23, 2009 at 17:30, Tony Chang t...@chromium.org wrote: Here's a crappy work around: Add a gclient hook that checks for grd file changes. When a grd file changes, force delete the header it would generate. I'm pretty sure this would prevent bad builds from IncrediBuild. Alternately, maybe we can make a reduced test case and file a bug against IncrediBuild. On Thu, Jul 23, 2009 at 4:44 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: Is it possible to force it to rebuild some files, or... I don't know, do you see some real way to fix this problem? On Thu, Jul 23, 2009 at 16:41, Tony Chang t...@chromium.org wrote: To elaborate on Peter's comment. IncrediBuild (which the buildbots use) get confused by changes to our grd files. Our grd files generate headers, which should then cause lots of cc files to get rebuilt. Visual Studio seems to always get this right, but IncrediBuild gets this wrong and cc files don't get rebuilt. I imagine IncrediBuild is checking the timestamp of the file before the headers are re-generated and doesn't realize it needs to rebuild. On Thu, Jul 23, 2009 at 4:38 PM, Peter Kastingpkast...@google.com wrote: On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.orgwrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
To elaborate on Peter's comment. IncrediBuild (which the buildbots use) get confused by changes to our grd files. Our grd files generate headers, which should then cause lots of cc files to get rebuilt. Visual Studio seems to always get this right, but IncrediBuild gets this wrong and cc files don't get rebuilt. I imagine IncrediBuild is checking the timestamp of the file before the headers are re-generated and doesn't realize it needs to rebuild. On Thu, Jul 23, 2009 at 4:38 PM, Peter Kastingpkast...@google.com wrote: On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
Is it possible to force it to rebuild some files, or... I don't know, do you see some real way to fix this problem? On Thu, Jul 23, 2009 at 16:41, Tony Chang t...@chromium.org wrote: To elaborate on Peter's comment. IncrediBuild (which the buildbots use) get confused by changes to our grd files. Our grd files generate headers, which should then cause lots of cc files to get rebuilt. Visual Studio seems to always get this right, but IncrediBuild gets this wrong and cc files don't get rebuilt. I imagine IncrediBuild is checking the timestamp of the file before the headers are re-generated and doesn't realize it needs to rebuild. On Thu, Jul 23, 2009 at 4:38 PM, Peter Kastingpkast...@google.com wrote: On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
Look at how the current gyp hook works. It looks for changes to .gyp files and only runs if a .gyp (and maybe gypi?) file has changed. You can find what header it generates by opening the grd file and parsing the XML (the XML lists the output files). You'll need to build the base directory (e.g., Debug/obj/global_intermediate/ and the Release version), I would just check for the Windows versions since we don't seem to have this problem with the mac or linux buildbots. On Thu, Jul 23, 2009 at 5:36 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: I think that this workaround may be worth it. I'm not familiar with the IncrediBuild, but I can help making the hook (and we can run it only on Windows). How do I make a hook know which grd files changed? And also know which headers it generates? Alternatively, maybe this Windows-only hook would just delete all generated headers (with a hardcoded list)? Generation seems to be cheap, and such hook seems trivial to write. So, yes, this hook is kludgey. But we are aware of its limitations, and it would eliminate one kind of build mysteries. What do you think? On Thu, Jul 23, 2009 at 17:30, Tony Chang t...@chromium.org wrote: Here's a crappy work around: Add a gclient hook that checks for grd file changes. When a grd file changes, force delete the header it would generate. I'm pretty sure this would prevent bad builds from IncrediBuild. Alternately, maybe we can make a reduced test case and file a bug against IncrediBuild. On Thu, Jul 23, 2009 at 4:44 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: Is it possible to force it to rebuild some files, or... I don't know, do you see some real way to fix this problem? On Thu, Jul 23, 2009 at 16:41, Tony Chang t...@chromium.org wrote: To elaborate on Peter's comment. IncrediBuild (which the buildbots use) get confused by changes to our grd files. Our grd files generate headers, which should then cause lots of cc files to get rebuilt. Visual Studio seems to always get this right, but IncrediBuild gets this wrong and cc files don't get rebuilt. I imagine IncrediBuild is checking the timestamp of the file before the headers are re-generated and doesn't realize it needs to rebuild. On Thu, Jul 23, 2009 at 4:38 PM, Peter Kastingpkast...@google.com wrote: On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
I think that this workaround may be worth it. I'm not familiar with the IncrediBuild, but I can help making the hook (and we can run it only on Windows). How do I make a hook know which grd files changed? And also know which headers it generates? Alternatively, maybe this Windows-only hook would just delete all generated headers (with a hardcoded list)? Generation seems to be cheap, and such hook seems trivial to write. So, yes, this hook is kludgey. But we are aware of its limitations, and it would eliminate one kind of build mysteries. What do you think? On Thu, Jul 23, 2009 at 17:30, Tony Chang t...@chromium.org wrote: Here's a crappy work around: Add a gclient hook that checks for grd file changes. When a grd file changes, force delete the header it would generate. I'm pretty sure this would prevent bad builds from IncrediBuild. Alternately, maybe we can make a reduced test case and file a bug against IncrediBuild. On Thu, Jul 23, 2009 at 4:44 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: Is it possible to force it to rebuild some files, or... I don't know, do you see some real way to fix this problem? On Thu, Jul 23, 2009 at 16:41, Tony Chang t...@chromium.org wrote: To elaborate on Peter's comment. IncrediBuild (which the buildbots use) get confused by changes to our grd files. Our grd files generate headers, which should then cause lots of cc files to get rebuilt. Visual Studio seems to always get this right, but IncrediBuild gets this wrong and cc files don't get rebuilt. I imagine IncrediBuild is checking the timestamp of the file before the headers are re-generated and doesn't realize it needs to rebuild. On Thu, Jul 23, 2009 at 4:38 PM, Peter Kastingpkast...@google.com wrote: On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: fighting the flakiness - resource bundle issues?
Here's a crappy work around: Add a gclient hook that checks for grd file changes. When a grd file changes, force delete the header it would generate. I'm pretty sure this would prevent bad builds from IncrediBuild. Alternately, maybe we can make a reduced test case and file a bug against IncrediBuild. On Thu, Jul 23, 2009 at 4:44 PM, Paweł Hajdan Jr.phajdan...@chromium.org wrote: Is it possible to force it to rebuild some files, or... I don't know, do you see some real way to fix this problem? On Thu, Jul 23, 2009 at 16:41, Tony Chang t...@chromium.org wrote: To elaborate on Peter's comment. IncrediBuild (which the buildbots use) get confused by changes to our grd files. Our grd files generate headers, which should then cause lots of cc files to get rebuilt. Visual Studio seems to always get this right, but IncrediBuild gets this wrong and cc files don't get rebuilt. I imagine IncrediBuild is checking the timestamp of the file before the headers are re-generated and doesn't realize it needs to rebuild. On Thu, Jul 23, 2009 at 4:38 PM, Peter Kastingpkast...@google.com wrote: On Thu, Jul 23, 2009 at 4:31 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: Some of the flaky failures are caused by resource bundle issues. If you are familiar with the build process, or the resource bundle, please take a look. It looks like something needed a manual clobber and didn't get it. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
