This is somewhat of a retrospective -- so please bear with me. I've had the
privilege of working on a clojure project for a couple of years now, and have
accumulated some 15-20k lines of clojure code. I'm taking a little time to
look back over what has worked for me and what hasn't in terms of code/project
organization -- and *I'd love to know what has worked for other people (or
hasn't)* for similarly large projects.
I knew my project was going to grow to at least as much code as it has now at
the start, and my domain problem was fairly well-defined. From the very
beginning I organized my code into many (15-20) different clojure 'projects'
using lein. Rather than organizing code into these projects *by function
area*, I found myself organizing code into projects by their *dependencies*.
So any code that used libraries X,Y,Z went into a project that declared those
dependencies -- even if a function made sense in a different namespace by name
-- if it needed deps that I already had in another project, I moved the
function to that project. For example, in the Java GIS world, if you do
anything with swing components, you can easily pull in 100s of MBs of
dependencies. My project involves GIS work both server-side rest-apis, but
it's also nice to pull up a quick swing component showing data on a map for
debugging etc. I don't have these things in the same project even though there
is some library overlap because the GUI deps are just too many.
I think for me on my project, the only reason to separate anything into
'projects' is for reuse based on dependencies -- i.e., to use a function/set of
functions I've written again, what's the minimal amount of deps to pull in.
It's worked well for me in that sense -- I'm able to create a new 'main'
project, include libraries that I want from my project, and not pull in 2,000
dependencies unless I need it. The dependencies I'm mostly concerned with here
are Java deps and not clojure deps. I've found a relatively small core set of
clojure deps I almost always want available to me (specter, timbre, core.async)
-- though even still I have a utility library project where I have a hard rule
of zero dependencies (for basic macros like (ignore-exception body-forms)).
I've used lein's checkouts and managed dependencies fairly successfully though
I still forget to lein install everything before I lein uberjar my final
delivery artifacts and end up debugging old code before realizing what happened
(and my way of installing everything is a bash for loop :-)).
I've found this approach somewhat tedious and have been wondering if there's a
better way -- and am very curious what others do.
What I've been playing around with lately is a different concept for my own
code organization:
- What if all my clojure code could go in one place, or one project? (Even if
it ended up being 20k+ lines of code)
- What if namespaces contained their required dependencies in their metadata?
- What if upon namespace creation, a namespace's dependencies were
automatically added to the classpath?
- What if functions declared in a namespace could also declare additional
dependencies? These would be added to the classpath upon first invocation of
the function. This is great for my seldom used functions that need many
dependencies -- code could live in the namespace that is matches its function
instead of squirreled away in a project just to match its deps.
I have written a basic library that does these things, and am currently trying
it out on a small scale. Concerns I've already been trying to address:
- Dynamically adding things to the classpath is generally considered a /bad/
thing to do.
- My code reads all pom.xmls on the classpath to determine what libraries
are already on the classpath -- and does not re-add libraries that are already
on the classpath. I think this alone takes care of a lot of issues with
dynamically adding deps to the classpath (multiple versions of libraries on the
classpath).
- Production deployments should not need to calculate classpaths dynamically
- This technique should be able to be used whether the classpath is
precomputed (i.e., for production), or dynamically during development.
Example namespace loaded deps:
(ns test
{:dependencies 'com.taoensso/timbre "4.10.0"}
(:require [taoensso.timbre :as timbre
:refer [info debug error warn spy]]))
-or-
(ns test
{:deps '{[[com.taoensso/timbre {:mvn/version "4.10.0"]]}}}
(:require [taoensso.timbre :as timbre
:refer [info debug error warn spy]]))
Now these namespace deps would be loaded dynamically by aliasing the ns macro
for development with one that loads deps dynamically. In production, the
metadata is simply attached to the namespace per normal use of the ns form, no
deps added dynamically.
(defn-deps test-fn-deps
"Test Function with optional deps"
{:dependencies '[[diffit "1.0.0"]