Hi Geert-Jan, On 25 Jun 2013, at 16:51, Geert Jan BEX wrote:
> Hi Easybuilders, > > A couple of questions: > • I'm running into trouble using run_cmd_qa in a custom easyblock. The > command's output is pretty huge, and after a while, the maximum hitCount is > exceeded. As I understand the inner workings of run_cmd_qa, it keeps on > reading bits of stdout/stderr from the command it is executing, and it tries > to match with the questions, answering them when found. Failing that, it > will check for matches in the no_qa list, and when finding one, it resets > hitCount, incrementing it otherwise. > So if I get this right, I've to provide sufficient no_qa elements so that the > whole of the output of the command is "covered", i.e., making sure that there > is a no_qa match very n lines? Your view on it is close to correct, but not fully. The hit count is only increased if the output remains the same and no question could be matched against the end of the output. As soon as the output changes, the hitCount is reset (if not, it's a bug), because progress was detected without needing to provide an answer. So, if some operation is taking a while, run_cmd_qa assumes it's a question (i.e., the command is waiting for input), and will try to match a question to the output. If it can't, it will die assuming something is wrong. So, basically, yes, you'll have to try and come up with no_qa entries (which can be regex patterns b.t.w.) that match any operation that is printed at the end of the output for a while, but is not an actual question (e.g. a compile or copy command). Up until now, we've never run into a procedure in which this proves impractical (i.e. usually the no_qa list can be kept quite small). Can you share an example of what you're running into? Also, out of curiosity: what is the custom easyblock for? > • There are a number of modules that are really basic, e.g., git, > CMake... Perhaps it would make sense to define a very minimal toolchain > (even one that simply uses the resident gcc) for these kind of modules? That > would allow to really minimize dependencies. Just a question, maybe it is a > dumb idea. That's one way, yes, but then you're depending on the system-provided tools again, which is one of the things we want to avoid. Encouraging to provide modules for all dependencies makes reproducibility more likely, both across different systems/OSs, and on the same system with one or multiple minor/major OS updates in between. Relying on whatever (version of) tools the system provides is too likely to change, which may result into builds that suddenly break (we've all seen that). That being said, more stuff should be part of compiler toolchains, yes. There is a ticket open for exactly that. Because tools/libs like zlib, ncurses, binutils (ld, ar, ...) are (very) commonly dependencies, they should be (and will be) part of the compiler toolchains we define. See https://github.com/hpcugent/easybuild-framework/issues/644 . Maybe also tar/gzip should be included. regards, Kenneth
