I noticed when I made that change that the ".dep" format is still not
documented anywhere. I'll document it soon.
Here's a first cut at the grammar of the value that is stored in a
".dep" file via `write`:
<deps> = (list <version-string>
(cons <hash-of-source> <hash-of-dependencies>)
<dep> ...)
<dep> = (cons 'indirect <direct-dep>)
| <direct-dep>
<direct-dep> = (cons 'ext <plain-dep>) ; non-module dependency
| <plain-dep> ; module dependency
<plain-dep> = (cons 'collects <byte-string>) ; a collection-relative path
| <byte-string> ; other path (not portable)
At Fri, 30 Jan 2015 22:29:11 -0800, Eric Dobson wrote:
> This change seemed to change the format of .dep files, likely as intended
> to add the indirect dependencies. Is there any documentation of what the
> format is supposed to be? Currently I've just been trying to read cm.rkt
> and understand how it treats them.
>
> On Thu, Jan 8, 2015 at 9:31 AM, <mfl...@racket-lang.org> wrote:
>
> > mflatt has updated `master' from c56c9250f1 to 95e85ec5bd.
> > http://git.racket-lang.org/plt/c56c9250f1..95e85ec5bd
> >
> > =====[ 2 Commits ]======================================================
> > Directory summary:
> > 45.1% pkgs/racket-doc/scribblings/raco/
> > 4.7% pkgs/racket-doc/scribblings/reference/
> > 47.5% racket/collects/compiler/
> >
> > ~~~~~~~~~~
> >
> > fe9a04d Matthew Flatt <mfl...@racket-lang.org> 2015-01-08 09:11
> > :
> > | doc tweaks for `raco {setup,make}`
> > :
> > M pkgs/racket-doc/scribblings/raco/make.scrbl | 4 ++--
> > M pkgs/racket-doc/scribblings/raco/setup.scrbl | 22
> > ++++++++++++----------
> >
> > ~~~~~~~~~~
> >
> > 95e85ec Matthew Flatt <mfl...@racket-lang.org> 2015-01-08 09:57
> > :
> > | add support for indirect CM dependencies; use in `lazy-require`
> > |
> > | If module M in package P imports module N from package Q,
> > | and if N has a `lazy-require` for a module in R that is
> > | triggered during the compilation of M, then P doesn't really
> > | depend on R; P depends on Q, and Q depends on R, and P
> > | shoudn't necessarily know anything about Q. At the same time,
> > | a change to the file in R means that M must be recompiled.
> > | So, continue to track the compilation dependency, but mark
> > | it as "indirect" so that the package-dependency checker can
> > | ignore the dependency.
> > :
> > M pkgs/racket-doc/scribblings/raco/make.scrbl | 33 ++++++++-----
> > M racket/collects/compiler/cm-accomplice.rkt | 14 +++---
> > M racket/collects/compiler/cm.rkt | 49
> > ++++++++++++++------
> > M racket/collects/racket/lazy-require.rkt | 2 +-
> > M racket/collects/setup/private/pkg-deps.rkt | 1 +
> > M .../racket-doc/scribblings/reference/syntax.scrbl | 8 ++--
> >
> > =====[ Overall Diff ]===================================================
> >
> > pkgs/racket-doc/scribblings/raco/make.scrbl
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/pkgs/racket-doc/scribblings/raco/make.scrbl
> > +++ NEW/pkgs/racket-doc/scribblings/raco/make.scrbl
> > @@ -123,7 +123,7 @@ would create only @filepath{compiled/b_rkt.zo} and
> >
> > @; ----------------------------------------------------------------------
> >
> > -@section{Dependency Files}
> > +@section[#:tag "Dependency Files"]{Dependency Files}
> >
> > In addition to a bytecode file, @exec{raco make} creates a file
> > @filepath{compiled/@nonterm{name}_@nonterm{ext}.dep} that records
> > @@ -538,7 +538,7 @@ messages are instances of a
> > @racket[parallel-compile-event] prefab structure:
> >
> > @racketblock[
> > (struct parallel-compile-event (worker event) #:prefab)
> > -].
> > +]
> >
> > The worker field is the index of the worker that the created the event.
> > The event
> > field is a @racket[compile-event] as document in
> > @@ -550,25 +550,36 @@ field is a @racket[compile-event] as document in
> >
> > @defmodule[compiler/cm-accomplice]
> >
> > -@defproc[(register-external-file [file (and path? complete-path?)])
> > void?]{
> > +@defproc[(register-external-file [file (and path? complete-path?)]
> > + [#:indirect? indirect? any/c #f])
> > + void?]{
> >
> > -Logs a message (see @racket[log-message]) at level @racket['info] to
> > -a logger named @racket['cm-accomplice]. The
> > -message data is a @racketidfont{file-dependency} prefab structure type
> > -with two fields; the first field's value is @racket[file] and the second
> > -field's value is @racket[#f] (to indicate a non-module dependency).
> > +Logs a message (see @racket[log-message]) at level @racket['info] to a
> > +logger named @racket['cm-accomplice]. The message data is a
> > +@racketidfont{file-dependency} prefab structure type with two fields;
> > +the first field's value is @racket[file] and the second field's value
> > +is @racket[#f] (to indicate a non-module dependency). If the
> > +@racket[indirect?] argument is true, the data is more specifically an
> > +instance of a @racketidfont{file-dependency/indirect} prefab structure
> > +type that is a subtype of @racketidfont{file-dependency} with no new
> > +fields.
> >
> > A compilation manager implemented by @racketmodname[compiler/cm] looks
> > -for such messages to register an external dependency. The compilation
> > -manager records (in a @filepath{.dep} file) the path as contributing
> > -to the implementation of the module currently being
> > +for such messages to register an external dependency. In response, the
> > +compilation manager records (in a @filepath{.dep} file) the path as
> > +contributing to the implementation of the module currently being
> > compiled. Afterward, if the registered file is modified, the
> > -compilation manager will know to recompile the module.
> > +compilation manager will know to recompile the module. An ``indirect''
> > +dependency has no effect on recompilation, but it can signal to other
> > +tools, such as a package-dependency checker, that the dependency is
> > +indirect (and should not imply a direct package dependency).
> >
> > The @racket[include] macro, for example, calls this procedure with the
> > path of an included file as it expands an @racket[include] form.}
> >
> > -@defproc[(register-external-module [file (and path? complete-path?)])
> > void?]{
> > +@defproc[(register-external-module [file (and path? complete-path?)]
> > + [#:indirect? indirect? any/c #f])
> > + void?]{
> >
> > Like @racket[register-external-file], but logs a message with a
> > @racketidfont{file-dependency} prefab structure type whose second
> >
> > pkgs/racket-doc/scribblings/raco/setup.scrbl
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/pkgs/racket-doc/scribblings/raco/setup.scrbl
> > +++ NEW/pkgs/racket-doc/scribblings/raco/setup.scrbl
> > @@ -744,13 +744,14 @@ Optional @filepath{info.rkt} fields trigger
> > additional actions by
> > module. More specifically, used modules are determined when
> > deleting a @filepath{.dep} file, which would have been created to
> > accompany a @filepath{.zo} file when the @filepath{.zo} was built
> > - by @exec{raco setup}. If the @filepath{.dep} file indicates another
> > - module, that module's @filepath{.zo} is deleted only if it also has
> > - an accompanying @filepath{.dep} file. In that case, the
> > - @filepath{.dep} file is deleted, and additional used modules are
> > - deleted based on the used module's @filepath{.dep} file, etc.
> > - Supplying a specific list of collections to @exec{raco setup} disables
> > - this dependency-based deletion of compiled files.}
> > + by @exec{raco setup} or @exec{raco make} (see
> > + @secref["Dependency\x20Files"]). If the @filepath{.dep} file
> > + indicates another module, that module's @filepath{.zo} is deleted
> > + only if it also has an accompanying @filepath{.dep} file. In that
> > + case, the @filepath{.dep} file is deleted, and additional used
> > + modules are deleted based on the used module's @filepath{.dep}
> > + file, etc. Supplying a specific list of collections to @exec{raco
> > + setup} disables this dependency-based deletion of compiled files.}
> >
> > ]
> >
> > @@ -816,9 +817,10 @@ with fewer dependencies.
> > @subsection{How Dependency Checking Works}
> >
> > Dependency checking uses @filepath{.zo} files, associated
> > -@filepath{.dep} files, and the documentation index. Dynamic
> > -references, such as through @racket[dynamic-require], are not visible
> > -to the dependency checker; only dependencies via @racket[require],
> > +@filepath{.dep} files (see @secref["Dependency Files"]), and the
> > +documentation index. Dynamic references, such as through
> > +@racket[dynamic-require], are not visible to the dependency checker;
> > +only dependencies via @racket[require],
> > @racket[define-runtime-module-path-index], and other forms that
> > cooperate with @racket[raco make] are visible for dependency checking.
> >
> >
> > pkgs/racket-doc/scribblings/reference/syntax.scrbl
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/pkgs/racket-doc/scribblings/reference/syntax.scrbl
> > +++ NEW/pkgs/racket-doc/scribblings/reference/syntax.scrbl
> > @@ -3005,10 +3005,10 @@ submodule). Introduced submodules have the names
> > @racket[lazy-require-]@racket[_n]@racketidfont{-}@racket[_m], where
> > @racket[_n] is a phase-level number and @racket[_m] is a number.
> >
> > -When the use of a lazily-required function triggers module loading,
> > -@racket[register-external-module] declares a potential compilation
> > -dependency (in case the function is used in the process of compiling a
> > -module).
> > +When the use of a lazily-required function triggers module loading, it
> > +also triggers a use of @racket[register-external-module] to declare an
> > +indirect compilation dependency (in case the function is used in the
> > +process of compiling a module).
> >
> > @examples[#:eval lazy-require-eval
> > (lazy-require
> >
> > racket/collects/compiler/cm-accomplice.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/compiler/cm-accomplice.rkt
> > +++ NEW/racket/collects/compiler/cm-accomplice.rkt
> > @@ -3,16 +3,18 @@
> > (provide register-external-file
> > register-external-module)
> >
> > -(define (register-external-file f)
> > - (register-external 'register-external-file f #f))
> > -(define (register-external-module f)
> > - (register-external 'register-external-module f #t))
> > +(define (register-external-file f #:indirect? [indirect? #f])
> > + (register-external 'register-external-file f #f indirect?))
> > +(define (register-external-module f #:indirect? [indirect? #f])
> > + (register-external 'register-external-module f #t indirect?))
> >
> > -(define (register-external who f module?)
> > +(define (register-external who f module? indirect?)
> > (unless (and (path? f) (complete-path? f))
> > (raise-type-error who "complete path" f))
> > (log-message (current-logger)
> > 'info
> > 'cm-accomplice
> > (format "file dependency: ~s" f)
> > - `#s(file-dependency ,f ,module?)))
> > + (if indirect?
> > + `#s((file-dependency/indirect file-dependency 2) ,f
> > ,module?)
> > + `#s(file-dependency ,f ,module?))))
> >
> > racket/collects/compiler/cm.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/compiler/cm.rkt
> > +++ NEW/racket/collects/compiler/cm.rkt
> > @@ -231,9 +231,13 @@
> > (define (get-dep-sha1s deps up-to-date collection-cache read-src-syntax
> > mode roots must-exist? seen)
> > (let ([l (for/fold ([l null]) ([dep (in-list deps)])
> > (and l
> > + ;; (cons 'indirect dep) => indirect dependency (for
> > pkg-dep checking)
> > ;; (cons 'ext rel-path) => a non-module file, check
> > source
> > ;; rel-path => a module file name, check cache
> > - (let* ([ext? (and (pair? dep) (eq? 'ext (car dep)))]
> > + (let* ([dep (if (and (pair? dep) (eq? 'indirect (car
> > dep)))
> > + (cdr dep)
> > + dep)]
> > + [ext? (and (pair? dep) (eq? 'ext (car dep)))]
> > [p (collects-relative*->path (if ext? (cdr dep)
> > dep) collection-cache)])
> > (cond
> > [ext? (let ([v (get-source-sha1 p)])
> > @@ -273,19 +277,26 @@
> > external-module-deps ; can
> > create cycles if misused!
> > reader-deps))]
> > [external-deps (remove-duplicates external-deps)])
> > + (define (path*->collects-relative/maybe-indirect dep)
> > + (if (and (pair? dep) (eq? 'indirect (car dep)))
> > + (cons 'indirect (path*->collects-relative (cdr dep)))
> > + (path*->collects-relative dep)))
> > (with-compile-output dep-path
> > (lambda (op tmp-path)
> > (let ([deps (append
> > - (map path*->collects-relative deps)
> > + (map path*->collects-relative/maybe-indirect deps)
> > (map (lambda (x)
> > - (cons 'ext (path*->collects-relative x)))
> > + (define d
> > (path*->collects-relative/maybe-indirect x))
> > + (if (and (pair? d) (eq? 'indirect d))
> > + (cons 'indirect (cons 'ext (cdr d)))
> > + (cons 'ext d)))
> > external-deps))])
> > - (write (list* (version)
> > - (cons (or src-sha1 (get-source-sha1 path))
> > - (get-dep-sha1s deps up-to-date
> > collection-cache read-src-syntax mode roots #t #hash()))
> > - deps)
> > - op)
> > - (newline op))))))
> > + (write (list* (version)
> > + (cons (or src-sha1 (get-source-sha1 path))
> > + (get-dep-sha1s deps up-to-date
> > collection-cache read-src-syntax mode roots #t #hash()))
> > + deps)
> > + op)
> > + (newline op))))))
> >
> > (define (format-time sec)
> > (let ([d (seconds->date sec)])
> > @@ -311,6 +322,7 @@
> > (define-struct ext-reader-guard (proc top)
> > #:property prop:procedure (struct-field-index proc))
> > (define-struct file-dependency (path module?) #:prefab)
> > +(define-struct (file-dependency/indirect file-dependency) () #:prefab)
> >
> > (define (compile-zo* mode roots path src-sha1 read-src-syntax zo-name
> > up-to-date collection-cache)
> > ;; The `path' argument has been converted to .rkt or .ss form,
> > @@ -322,10 +334,14 @@
> > (define reader-deps null)
> > (define deps-sema (make-semaphore 1))
> > (define done-key (gensym))
> > - (define (external-dep! p module?)
> > + (define (external-dep! p module? indirect?)
> > + (define bstr (path->bytes p))
> > + (define dep (if indirect?
> > + (cons 'indirect bstr)
> > + bstr))
> > (if module?
> > - (set! external-module-deps (cons (path->bytes p)
> > external-module-deps))
> > - (set! external-deps (cons (path->bytes p) external-deps))))
> > + (set! external-module-deps (cons dep external-module-deps))
> > + (set! external-deps (cons dep external-deps))))
> > (define (reader-dep! p)
> > (call-with-semaphore
> > deps-sema
> > @@ -386,7 +402,8 @@
> > (file-dependency? (vector-ref l 2))
> > (path? (file-dependency-path (vector-ref l 2))))
> > (external-dep! (file-dependency-path (vector-ref l 2))
> > - (file-dependency-module? (vector-ref l 2))))
> > + (file-dependency-module? (vector-ref l 2))
> > + (file-dependency/indirect? (vector-ref l 2))))
> > (loop))))
> >
> > ;; Write the code and dependencies:
> > @@ -627,9 +644,13 @@
> > ;; If `sha1-only?', then `maybe-compile-zo' returns a #f or
> > thunk:
> > (maybe-compile-zo sha1-only? deps mode roots path orig-path
> > read-src-syntax up-to-date collection-cache new-seen)]
> > [(ormap
> > - (lambda (p)
> > + (lambda (raw-p)
> > + ;; (cons 'indirect dep) => indirect dependency (for
> > pkg-dep checking)
> > ;; (cons 'ext rel-path) => a non-module file (check date)
> > ;; rel-path => a module file name (check transitive
> > dates)
> > + (define p (if (and (pair? raw-p) (eq? 'indirect (car
> > raw-p)))
> > + (cdr raw-p)
> > + raw-p))
> > (define ext? (and (pair? p) (eq? 'ext (car p))))
> > (define d (collects-relative*->path (if ext? (cdr p) p)
> > collection-cache))
> > (define t
> >
> > racket/collects/racket/lazy-require.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/racket/lazy-require.rkt
> > +++ NEW/racket/collects/racket/lazy-require.rkt
> > @@ -110,4 +110,4 @@
> > modpath
> > (variable-reference->resolved-module-path vr))))])
> > (when (path? path)
> > - (register-external-module path))))
> > + (register-external-module path #:indirect? #t))))
> >
> > racket/collects/setup/private/pkg-deps.rkt
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > --- OLD/racket/collects/setup/private/pkg-deps.rkt
> > +++ NEW/racket/collects/setup/private/pkg-deps.rkt
> > @@ -489,6 +489,7 @@
> > ;; Treat everything in ".dep" as 'build mode...
> > (define deps (cddr (call-with-input-file* (build-path dir f)
> > read)))
> > (for ([dep (in-list deps)])
> > + ;; Note: indirect dependencies (which start with 'indirect)
> > are ignored
> > (when (and (pair? dep)
> > (eq? 'collects (car dep)))
> > (define path-strs (map bytes->string/utf-8 (cdr dep)))
> >
> _________________________
> Racket Developers list:
> http://lists.racket-lang.org/dev
_________________________
Racket Developers list:
http://lists.racket-lang.org/dev
