This is somewhat of a retrospective -- so please bear with me.  I've had the 
privilege of working on a clojure project for a couple of years now, and have 
accumulated some 15-20k lines of clojure code.  I'm taking a little time to 
look back over what has worked for me and what hasn't in terms of code/project 
organization -- and *I'd love to know what has worked for other people (or 
hasn't)* for similarly large projects.

I knew my project was going to grow to at least as much code as it has now at 
the start, and my domain problem was fairly well-defined.  From the very 
beginning I organized my code into many (15-20) different clojure 'projects' 
using lein.  Rather than organizing code into these projects *by function 
area*, I found myself organizing code into projects by their *dependencies*.  
So any code that used libraries X,Y,Z went into a project that declared those 
dependencies -- even if a function made sense in a different namespace by name 
-- if it needed deps that I already had in another project, I moved the 
function to that project.  For example, in the Java GIS world, if you do 
anything with swing components, you can easily pull in 100s of MBs of 
dependencies.  My project involves GIS work both server-side rest-apis, but 
it's also nice to pull up a quick swing component showing data on a map for 
debugging etc.  I don't have these things in the same project even though there 
is some library overlap because the GUI deps are just too many.

I think for me on my project, the only reason to separate anything into 
'projects' is for reuse based on dependencies -- i.e., to use a function/set of 
functions I've written again, what's the minimal amount of deps to pull in.  
It's worked well for me in that sense -- I'm able to create a new 'main' 
project, include libraries that I want from my project, and not pull in 2,000 
dependencies unless I need it.  The dependencies I'm mostly concerned with here 
are Java deps and not clojure deps.  I've found a relatively small core set of 
clojure deps I almost always want available to me (specter, timbre, core.async) 
-- though even still I have a utility library project where I have a hard rule 
of zero dependencies (for basic macros like (ignore-exception body-forms)).

I've used lein's checkouts and managed dependencies fairly successfully though 
I still forget to lein install everything before I lein uberjar my final 
delivery artifacts and end up debugging old code before realizing what happened 
(and my way of installing everything is a bash for loop :-)).

I've found this approach somewhat tedious and have been wondering if there's a 
better way -- and am very curious what others do.

What I've been playing around with lately is a different concept for my own 
code organization:
  - What if all my clojure code could go in one place, or one project? (Even if 
it ended up being 20k+ lines of code)
  - What if namespaces contained their required dependencies in their metadata?
  - What if upon namespace creation, a namespace's dependencies were 
automatically added to the classpath?
  - What if functions declared in a namespace could also declare additional 
dependencies?  These would be added to the classpath upon first invocation of 
the function.  This is great for my seldom used functions that need many 
dependencies -- code could live in the namespace that is matches its function 
instead of squirreled away in a project just to match its deps.

I have written a basic library that does these things, and am currently trying 
it out on a small scale.  Concerns I've already been trying to address:
  - Dynamically adding things to the classpath is generally considered a /bad/ 
thing to do.
    - My code reads all pom.xmls on the classpath to determine what libraries 
are already on the classpath -- and does not re-add libraries that are already 
on the classpath.  I think this alone takes care of a lot of issues with 
dynamically adding deps to the classpath (multiple versions of libraries on the 
classpath).
  - Production deployments should not need to calculate classpaths dynamically
    - This technique should be able to be used whether the classpath is 
precomputed (i.e., for production), or dynamically during development.

Example namespace loaded deps:
(ns test
  {:dependencies '[[[[com.taoensso/timbre "4.10.0"]]]]}
  (:require [taoensso.timbre :as timbre
             :refer [info debug error warn spy]]))
-or-
(ns test
  {:deps '{[[com.taoensso/timbre {:mvn/version "4.10.0"]]}}}
  (:require [taoensso.timbre :as timbre
             :refer [info debug error warn spy]]))

Now these namespace deps would be loaded dynamically by aliasing the ns macro 
for development with one that loads deps dynamically.  In production, the 
metadata is simply attached to the namespace per normal use of the ns form, no 
deps added dynamically.

(defn-deps test-fn-deps
     "Test Function with optional deps"
     {:dependencies '[[diffit "1.0.0"]
                      [com.taoensso/timbre "4.10.0"]]
      :require '([diffit.vec :as d]
                 [taoensso.timbre :as timbre
                  :refer [log  trace  debug  info  warn  error  fatal  report
                          logf tracef debugf infof warnf errorf fatalf reportf
                          spy get-env]])
      :import '((java.util HashMap)
                (java.io InputStream))}
     [x y]
     (info :hello)
     (d/diff x y))

Using defn-deps is /not/ transparent, as it does requires and imports.  This I 
think is ok though, for a couple of reasons:
  - If the deps are already on the classpath (production), no changes, and if 
they aren't, you get an exception (this is good -- it's an optional function 
that you chose not to include its deps)
  - These functions should be pretty rare -- they are really functions that are 
useful if certain optional dependencies are on the classpath.

Thoughts?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to