On Tue, 2020-04-21 at 21:51 -0500, Joshua Watt wrote: > On Sun, Apr 19, 2020 at 12:04 PM Richard Purdie > <richard.pur...@linuxfoundation.org> wrote: > Continuing the discussion from the OE TSC meeting, now that I've had > a little time to digest the idea: I don't think the filter functions > are necessarily a bad idea. I think the real problem is how to > prevent them from becoming API maintenance and debugging problems, > and as you stated, performance issues. I think there might be a few > ways to address this, and most of it comes down to strictly defining > the API. > Firstly, I don't think we want to allow (at least at this point) > completely arbitrary python functions to be called as a filter. > Hopefully, we can figure out the few filter manipulations that are > required and design the API such that those can be used without > having to open it all the way up.
I think that is a sensible idea. The good news is we know roughly what the current needs look like, the code is in lib/oe/classextend.py: def extend_name(self, name): if name.startswith("kernel-") or name == "virtual/kernel": return name if name.startswith("rtld"): return name if name.endswith("-crosssdk"): return name if name.endswith("-" + self.extname): name = name.replace("-" + self.extname, "") if name.startswith("virtual/"): subs = name.split("/", 1)[1] if not subs.startswith(self.extname): return "virtual/" + self.extname + "-" + subs return name if name.startswith("/") or (name.startswith("${") and name.endswith("}")): return name if not name.startswith(self.extname): return self.extname + "-" + name return name def map_depends(self, dep): if dep.endswith(("-native", "-native-runtime")) or ('nativesdk-' in dep) or ('cross-canadian' in dep) or ('-crosssdk-' in dep): return dep else: # Do not extend for that already have multilib prefix var = self.d.getVar("MULTILIB_VARIANTS") if var: var = var.split() for v in var: if dep.startswith(v): return dep return self.extend_name(dep) which apart from one getVar call, is all static string manipulation. Could be better, could be worse. Its surprisingly hard to manipulate a depends string (we also have to use the explode_deps function as the strings can have versions in). > Secondly, I think we should assume that any > filter function is "pure" [1], meaning it only takes in the input > variable value string, possibly some other manipulation arguments, > and outputs the new variable value string. Notable, it should *not* > take in the datastore object, since this is inherently global data > that can change at any time. Hopefully, this will allow optimizations > and prevent performance issues because bitbake can keep a cache of > filter function inputs mapped to outputs. Since the function outputs > *only* depend on the inputs, the filter can avoid being called > multiple times if the input set has been seen before. That definitely looks like a good move and alleviates some of the concerns I have about it. I do wonder how we handle MULTILIB_VARIANTS in the above example. Perhaps some kind of cut down key/value pairs or allow parameters? > I can dream up a few ways this might look. In this example, I'm > defining a filter function that simply adds a defined prefix to every > value in a space separated variable. > > First, the filter function itself: > > def filter_prefix(in, prefix): > return ' '.join(prefix + v for v in in.split()) > > As for how to express this in a recipe, I can think of a few ways. > This first one is a variable flag. The base flag is "filter" and the > suffix indicates which filter is applied (in this case, the "prefix" > function). The value are the extra arguments passed to the function. > > DEPENDS[filter:prefix] = "${MLPREFIX}" > > Another option would be to encode the function and arguments in the > flag value, which might also allow filters to be chained (although, I > would hope that's a rare requirement!): > > DEPENDS[filter] = "prefix ${MLPREFIX}" > > Chaining might look something like: > > DEPENDS[filter] = "prefix ${MLPREFIX}; suffix ${MY_SUFFIX}" > > The great part about pure functions here is that the property is > transitive, so the entire result of chain with both operations (e.g. > suffix(prefix(${DEPENDS}))) can be cached if desired. Those are good suggestions. Thinking out loud, I'm kind of drawn to something which looks a bit more function like: DEPENDS[filter] = "prefix_filter('${MLPREFIX}') suffix_filter('${MY_SUFFIX}')" as it might be more obvious and searchable through grep? I don't like ";" as a delimiter as it doesn't work well with append/prepend of values. Its a tradeoff between the need for some parameters to be passed in and complexity. I do like the fact we can cache this approach, that will help a lot with performance. > There's probably a few other ways to express this, but the idea is > that it's more tightly controlled about what can be applied. > > [1]: https://en.wikipedia.org/wiki/Pure_function Its definitely the way to go... Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1058): https://lists.openembedded.org/g/openembedded-architecture/message/1058 Mute This Topic: https://lists.openembedded.org/mt/73131657/21656 Group Owner: openembedded-architecture+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-