On Tue, Sep 24, 2019 at 04:42:01PM -0300, Willian Rampazzo wrote: > Hello everyone, > > I started working on Trello card > https://trello.com/c/T3SC1sZs/1521-implement-fetch-assets-command-line > as part of a broader card, > https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher > and I would like to bring my findings to a discussion. > > One way to implement that would be to parse the test source looking > for the fetch_asset call and execute it. In theory, it is a straight > forward implementation. > > I have started working in the parser. After some simple tests, some > complex situations started to show up. Let me bring an existing > example, examples/tests/assets.py: > > def setUp(self): > mirrors = ['https://mirrors.peers.community/mirrors/gnu/hello/', > 'https://mirrors.kernel.org/gnu/hello/', > 'http://gnu.c3sl.ufpr.br/ftp/', > 'ftp://ftp.funet.fi/pub/gnu/prep/hello/'] > hello = 'hello-2.9.tar.gz' > hello_locations = ["%s/%s" % (loc, hello) for loc in mirrors] > hello_sig = 'hello-2.9.tar.gz.sig' > hello_sig_locations = ["%s/%s" % (loc, hello_sig) for loc in mirrors] > self.hello = self.fetch_asset( > name=hello, > locations=hello_locations) > self.hello_sig = self.fetch_asset( > name=hello_sig, > asset_hash='f3b9fae20c35740004ae7b8de1301836dab4ac30', > locations=hello_sig_locations) > > When the parser finds the fetch_asset call, it needs to inspect its > arguments, looking for variables. If any variable is found, it needs > to walk back the code looking for its assignment, so it is able to > execute it prior to the fetch_asset execution. If the variables > consist of other variables, the parser, again, needs to walk back in > the code looking for those assignments, and so on. > > There are countless ways of creating the right side of a variable > assignment, from a single string assignment to a function that builds > the content, or conditional assignments. > > My initial idea is to cover variables consisting of other variables > and list comprehension, just like in the example. For now, other > complex constructions would be out of this implementation. >
I would actually start with something even simpler, that is, only consider the fetch_asset() calls with literals as the `name` parameter, and with `name` being a URL (that is, no locations are given or considered). If that works, we can have an example for a tool that checks/downloads those, and increase the coverage later (to include, as you put it "variables consisting of other variables and list comprehension"). > Any comment, concern, suggestion, is appreciated here as it would help > to build a more robust code. > I consider this type of parsing a kind of "magic" (hopefully the good kind). I'd try the "bunny from a hat" trick before attempting the "sawing a human" in half trick. Cheers, - Cleber. > Best regards, > > Willian Rampazzo > Software Engineer > Red Hat Brazil > 2717 337F 7E4A 5FDF >
