Yes, the problem is that I have lots of generated files. So parsing the
files is needed both to identify the outputs (which tup requires) and to
identify the inputs, since I don't want to have to specify all the inputs
manually.
As an example of what I'm working with, I'm attaching a couple of my
run-scripts. One handles building a latex file, and the other generates
rules for building pdf files from python scripts, that themselves read in
data files. The rules to generate these data files are separate.
A third script handles building haskell files. In this case, every module
depends on ".hi" files generated by other builds. So "generated sources"
is built into the language, and one cannot rely on tup's built-in
dependency finder.
All three of these are very quick and dirty scripts that are specific to my
particular scientific code. But none of them would I be very happy
implementing in lua. And in the matplotlib parser, I do indeed use the
eval function, so even though it's just a hokey regular expression munger,
it handles (some) python code in a way that would be very much more
difficult in lua.
In general, when thinking about scripting, I recommend keeping the case of
a non-trivial build in mind. And if the lua-based scripting doesn't help
much with the case of a non-trivial build, then it might not be worth its
weight.
David
On Mon, Nov 10, 2014 at 9:23 AM, Spooge <[email protected]> wrote:
>
> On 11/09/2014 12:04 AM, David Roundy wrote:
>
>> But my primary reason for wanting scripting is to be able to automate the
>> actual creation of dependency relationships.
>>
> Tup should automatically detect file dependencies of source files without
> a language-specific parser. My impression was that this was what CMake's
> file parsers are for. You may already know this though; I may be
> underestimating the complexity of your situation.
>
> Many languages have existing libraries that would be effective for this
>> sort of task, which is what makes run-script so great.
>>
> Integrating a language directly in the Tup binary solves a lot of the
> complexities of using run-scripts, but with the obvious caveat that you're
> limited to the language that is integrated.
>
> If it would help though, it should be possible to enable Lua's require
> keyword for including external modules. I disabled it because I didn't
> know how to track module dependencies, dealing with dependencies outside
> the project root, etc. but including common parsers would be a very good
> use case.
>
>
> --
> --
> tup-users mailing list
> email: [email protected]
> unsubscribe: [email protected]
> options: http://groups.google.com/group/tup-users?hl=en
> --- You received this message because you are subscribed to a topic in the
> Google Groups "tup-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/tup-users/Hl00qYKjq7c/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
David Roundy
--
--
tup-users mailing list
email: [email protected]
unsubscribe: [email protected]
options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups
"tup-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
#!/usr/bin/python2
import glob, re, string, sys
texfs = glob.glob('*.tex')
if len(sys.argv) > 1:
texfs = sys.argv[1:]
documentclassre = re.compile(r'\\documentclass(\[[^\]]+\])?{')
graphicre = re.compile(r'^\s*\\includegraphics(\[[^\]]+\])?{([^}]+)}', re.MULTILINE)
inputre = re.compile(r'\\input(\[[^\]]+\])?{([^}]+)}')
for fname in texfs:
f = open(fname, 'r')
latex = f.read()
if documentclassre.search(latex):
pdfs = [x[1]+'.pdf' for x in graphicre.findall(latex)]
texs = [x[1]+'.tex' for x in inputre.findall(latex)]
if len(pdfs + texs) > 0:
print ': %s | %s |> !latex |>' % (fname, string.join(pdfs + texs))
else:
print ': %s |> !latex |>' % (fname)
else:
print 'no good', fname
#!/usr/bin/python2
import re, glob, string, os, sys
import_re = re.compile(r"^import\s+(\w+)", re.M)
fixed_open = re.compile(r"=\s*open\(['\"]([^'\"]*)['\"]\s*,\s*['\"]w['\"]\s*\)")
fixed_output = re.compile(r"savefig\(['\"]([^'\"]*)['\"](\s*,[\w\s=]+)*\)")
changing_output = re.compile(r"savefig\(['\"]([^'\"]*)['\"]\s*%\s*(\(.*\))(\s*,[\w\s=]+)*\s*\)")
arguments = re.compile(r"^#arg\s+(\w+)\s*=\s*(.*)$", re.M)
fixed_open_input = re.compile(r"=\s*open\(['\"]([^'\"]*)['\"]\s*,\s*['\"]r['\"]\s*\)")
fixed_input = re.compile(r"^[^\n#]*loadtxt\(['\"]([^'\"]*)['\"]\)", re.M)
changing_loadtxt_noparens = re.compile(r"^[^\n#]*loadtxt\(['\"]([^'\"]*)['\"]\s*%\s*([^\(\)\n]*)(\s*,[\w\s=]+)*\s*\)", re.M)
changing_loadtxt = re.compile(r"^[^\n#]*loadtxt\(['\"]([^'\"]*)['\"]\s*%\s*(\([^\)]*\))(\s*,[\w\s=]+)*\s*\)", re.M)
changing_input = re.compile(r"input:\s*['\"]([^'\"]*)['\"]\s*%\s*(\(.*\))")
list_comprehension_input = re.compile(r"input:\s*(\[.+for.*in.*\])\n")
def printrule(pyf, args, inputs, outputs):
if len(outputs) == 0:
return
if len(inputs) == 0:
print ': %s | *.pyc |> cd .. && python2 figs/%s %s |> %s' % (pyf, pyf, args, string.join(outputs))
return
print ': %s | *.pyc %s |> cd .. && python2 figs/%s %s |> %s' % (pyf, string.join(inputs), pyf, args, string.join(outputs))
def friendly_eval(code, context, local = None):
try:
return eval(code, local)
except:
print ("\nError evaluating '%s' from file %s" % (code, context))
raise
pyfs = glob.glob('*.py')
if len(sys.argv) > 1:
pyfs = sys.argv[1:]
for pyf in pyfs:
f = open(pyf, 'r')
contents = f.read()
f.close()
inputs = fixed_input.findall(contents)
inputs += fixed_open_input.findall(contents)
inputs = [i[5:] for i in inputs]
imports = import_re.findall(contents)
for i in imports:
ipath = i+'.py'
if os.path.exists(ipath):
inputs.append(ipath)
outputs = fixed_output.findall(contents)
outputs = [x[0] for x in outputs]
outputs += fixed_open.findall(contents) # add any files created with open(...)
outputs = [o[5:] for o in outputs]
argvals = arguments.findall(contents)
if len(argvals) > 0:
coutputs = changing_output.findall(contents)
list_inputs = list_comprehension_input.findall(contents)
cinputs = changing_input.findall(contents)
cinputs += changing_loadtxt.findall(contents)
cinputs += changing_loadtxt_noparens.findall(contents)
allvalues = [{}]
commandlineformat = ""
commandlineargs = []
for arg in argvals:
commandlineformat += " \"%s\""
commandlineargs.append(arg[0])
newallvalues = []
for v in friendly_eval(arg[1], pyf):
for av in allvalues:
newav = av.copy()
newav[arg[0]] = v
newallvalues.append(newav)
allvalues = newallvalues
for a in allvalues:
aa = commandlineformat % friendly_eval('(' + string.join(commandlineargs, ",") + ')', 'file '+ pyf, a)
extrainputs = []
extraoutputs = []
for i in list_inputs:
#print 'examining input:', i
fnames = friendly_eval(i, pyf, a)
#print 'found', fnames
extrainputs += fnames
for i in cinputs:
#print 'examining input:', i[0], '%', '"%s"' % i[1]
fname = i[0] % friendly_eval(i[1], pyf, a)
extrainputs.append(fname)
for o in coutputs:
fname = o[0] % friendly_eval(o[1], pyf, a)
#print 'generate', fname, 'from', aa
extraoutputs.append(fname)
extrainputs = [e[5:] for e in extrainputs]
extraoutputs = [e[5:] for e in extraoutputs]
printrule(pyf, aa, inputs + extrainputs, outputs + extraoutputs)
else:
printrule(pyf, "", inputs, outputs)
#!/usr/bin/python
import glob, re, string
importre = re.compile('^import\s+(\S+)', re.MULTILINE)
mainre = re.compile('^main\s+=', re.MULTILINE)
hsfiles = glob.glob('*.hs')
imports = {}
mainfiles = []
allobjects = [hsf[:-2]+'o' for hsf in hsfiles]
allhi = [hsf[:-2]+'hi' for hsf in hsfiles]
for hsf in hsfiles:
f = open(hsf, 'r')
hs = f.read()
f.close()
if mainre.search(hs):
mainfiles.append(hsf)
imports[hsf] = importre.findall(hs)
imports[hsf] = [i for i in imports[hsf] if i+'.hs' in hsfiles]
done = {}
while len(hsfiles) > 0:
for hsf in hsfiles:
isokay = True
for i in imports[hsf]:
if i+'.hs' in hsfiles:
isokay = False
break
if isokay:
print ': %s | %s |> !ghc |>' % (hsf, string.join([i + '.hi' for i in imports[hsf]]))
hsfiles.remove(hsf)
allobjects = string.join(allobjects + allhi)
for main in mainfiles:
print ': %s | %s |> !ghclink |>' % (main, allobjects)
