Re: Taskcluster log fetching
On Thu, Oct 17, 2019 at 12:59 PM Tom Ritter wrote: > I wrote a similar thing, not nearly as friendly, that takes a taskgroupid: > https://gist.github.com/tomrittervg/9e99de9b3c517b8ba4e87d2a86985616 > > It seems like there should be some better platform for communicating these > types of tools. > While it requires a level of polish that not everyone has time for, I'd encourage developers who have scripts like these to think about how they might be integrated into `mach`. Either as a top-level command, a subcommand, or even just behind a flag on an existing command. If you think you have something that would be useful to a wider audience, I'm happy to provide feedback on where / how it can be integrated. Even if you don't have time, getting a bug on file (with a link to your repo) would be helpful. Moving it into mach might even make a good contributor project. For instance, in this case a more general purpose `mach taskgraph fetch-artifacts` command could be useful. There are already helper functions to download artifacts in-tree, so no scraping required. -Andrew > -tom > > PS: Other gists I have: > https://gist.github.com/tomrittervg/90329fea97486bfc9e4e8a4c8d51cc06 - > parse the output of MOZ_LOG="nsStandardURL:5" > https://gist.github.com/tomrittervg/adb8688426a9a5340da96004e2c8af79 - for > MMLog:5 > https://gist.github.com/tomrittervg/3cc0f08995a7cb71a2c66317204c0340 - for > searching through my old try pushes looking for something specific > https://gist.github.com/tomrittervg/ea852876f657ff7ca8684b2415be694f - for > printing a stack with MOZ_LOG > All of these are ugly half-developed one-offs but may be illustrative. > > -tom > > On Thu, Oct 17, 2019 at 4:43 PM Steve Fink wrote: > > > I wrote another mini tool for my own purposes that I felt might be of > > use to more people. > > > > Periodically, I want to gather some sort of data from nontrivial browser > > runs. So I'll stick some printfs into the code and do a try push -- most > > often to run Talos jobs, since I naively think of them as best > > representing "typical" browser usage. That injects my printouts into a > > whole mess of log files, which I can then go clickety click click for > > half an hour to download individually. (In practice, I'm lazy, so I > > download 3-4 and start working on processing them, then never get around > > to grabbing the ones from the other jobs. Thus wasting 90% of the > > resources burned by the push.) > > > > So now I have a `get-taskcluster-logs` script that will pull them all > > down into a directory in one go: > > > > get-taskcluster-logs -r > > > > or > > > > get-taskcluster-logs '' > > > > Much nicer. It is now much, much faster to get to the point where I kick > > myself for not injecting the pid or some other identifier in my log > > messages, because I notice that all of the different processes' outputs > > are smushed together. But that's another story. > > > > For details, see > > https://github.com/hotsphink/sfink-tools/blob/master/README.md or just > > grab the script directly from > > > > > https://raw.githubusercontent.com/hotsphink/sfink-tools/master/bin/get-taskcluster-logs > > > > Or checkout the whole repo at > > > > (hg) https://hg.sr.ht/~sfink/sfink-tools > > > > (git) https://github.com/hotsphink/sfink-tools/ > > > > > > ___ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Taskcluster log fetching
I wrote a similar thing, not nearly as friendly, that takes a taskgroupid: https://gist.github.com/tomrittervg/9e99de9b3c517b8ba4e87d2a86985616 It seems like there should be some better platform for communicating these types of tools. -tom PS: Other gists I have: https://gist.github.com/tomrittervg/90329fea97486bfc9e4e8a4c8d51cc06 - parse the output of MOZ_LOG="nsStandardURL:5" https://gist.github.com/tomrittervg/adb8688426a9a5340da96004e2c8af79 - for MMLog:5 https://gist.github.com/tomrittervg/3cc0f08995a7cb71a2c66317204c0340 - for searching through my old try pushes looking for something specific https://gist.github.com/tomrittervg/ea852876f657ff7ca8684b2415be694f - for printing a stack with MOZ_LOG All of these are ugly half-developed one-offs but may be illustrative. -tom On Thu, Oct 17, 2019 at 4:43 PM Steve Fink wrote: > I wrote another mini tool for my own purposes that I felt might be of > use to more people. > > Periodically, I want to gather some sort of data from nontrivial browser > runs. So I'll stick some printfs into the code and do a try push -- most > often to run Talos jobs, since I naively think of them as best > representing "typical" browser usage. That injects my printouts into a > whole mess of log files, which I can then go clickety click click for > half an hour to download individually. (In practice, I'm lazy, so I > download 3-4 and start working on processing them, then never get around > to grabbing the ones from the other jobs. Thus wasting 90% of the > resources burned by the push.) > > So now I have a `get-taskcluster-logs` script that will pull them all > down into a directory in one go: > > get-taskcluster-logs -r > > or > > get-taskcluster-logs '' > > Much nicer. It is now much, much faster to get to the point where I kick > myself for not injecting the pid or some other identifier in my log > messages, because I notice that all of the different processes' outputs > are smushed together. But that's another story. > > For details, see > https://github.com/hotsphink/sfink-tools/blob/master/README.md or just > grab the script directly from > > https://raw.githubusercontent.com/hotsphink/sfink-tools/master/bin/get-taskcluster-logs > > Or checkout the whole repo at > > (hg) https://hg.sr.ht/~sfink/sfink-tools > > (git) https://github.com/hotsphink/sfink-tools/ > > > ___ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Taskcluster log fetching
I wrote another mini tool for my own purposes that I felt might be of use to more people. Periodically, I want to gather some sort of data from nontrivial browser runs. So I'll stick some printfs into the code and do a try push -- most often to run Talos jobs, since I naively think of them as best representing "typical" browser usage. That injects my printouts into a whole mess of log files, which I can then go clickety click click for half an hour to download individually. (In practice, I'm lazy, so I download 3-4 and start working on processing them, then never get around to grabbing the ones from the other jobs. Thus wasting 90% of the resources burned by the push.) So now I have a `get-taskcluster-logs` script that will pull them all down into a directory in one go: get-taskcluster-logs -r or get-taskcluster-logs '' Much nicer. It is now much, much faster to get to the point where I kick myself for not injecting the pid or some other identifier in my log messages, because I notice that all of the different processes' outputs are smushed together. But that's another story. For details, see https://github.com/hotsphink/sfink-tools/blob/master/README.md or just grab the script directly from https://raw.githubusercontent.com/hotsphink/sfink-tools/master/bin/get-taskcluster-logs Or checkout the whole repo at (hg) https://hg.sr.ht/~sfink/sfink-tools (git) https://github.com/hotsphink/sfink-tools/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
