time ec2-describe-images -a > ec2-cli-images.txt real 1m26.401s user 0m6.551s sys 0m1.159s
and writes a 7.5MB file to disk. Note the -a flag, to list all of the available public images. in a repl, (time (spit "clj-awz-images.txt" (describe-images))) "Elapsed time: 90258.47 msecs" and writes an 18MB file to disk containing all the available public images. Am I missing something? You can also pass a list of filters to the call to narrow the result. On Friday, March 28, 2014 7:59:48 AM UTC-7, Dave Tenny wrote: > > I'm trying to code some amazonica based solutions in a nontrivial AWS > environment. > I work with many AWS accounts and it isn't unusual to see a thousand > instances running on one account, and similar excesses in other types of > AWS resources. So if you're going an ec2-describe-instances (or amazonica > equivalent), it needs not to choke in this environment. > > I like the way amazonica does all the bean marshalling for me so I can > express queries simply. But the returned datasets need to be more > pragmatic/performant. > > The problem for me is that Amazonica doesn't seem up to the task of > dealing with queries that return large volumes of data. > It has nothing to do with reflection I suspect, and more to do with > unwieldy amounts of duplicate information in the result unmarshalling > process. > The "clojure all the way down" philosophy results of duplicated > information and just printing the result to a file takes a long time. > If I accidentally let the output go to an emacs cider repl buffer, then > things get so wedged up to the point I may as well kill -9 emacs. > (Known cider repl issues here, it isn't all amazonica). > > For example: here's how long it takes to run the java based ec2 cli to > describe instances on an account: > > $ time ec2-describe-images >/tmp/ec2-cli-images.out > > real 0m11.484s > user 0m2.564s > sys 0m0.129s > > > And here's how long it takes from a 'lein repl' to run the same query on > the same account: > > (time (with-output ["/tmp/clj-awz-images.out"] (println > (ec2/describe-images)))) > "Elapsed time: 194685.552683 msecs" > > Now the amount of data being printed by the EC2 CLI is of course much > different than the output from Amazonica, > amazonica is returning everything in gory duplicate map detail, ec2 is > not, as evidenced by the relative output sizes: > > -rw-rw-r--. 1 dave dave 17201290 Mar 28 10:35 clj-awz-images.out > -rw-rw-r--. 1 dave dave 99342 Mar 28 10:26 ec2-cli-images.out.11.5s > > Where the amazonica output starts with: > {:images [{:hypervisor xen, :state available, :virtualization-type > paravirtual, :root-device-type instance-store, > ... and goes on like that with duplicate keywords all the way down. > > Anyway, my goal isn't to turn amazonica into ec2 cli. But even the most > trivial operations in amazonica (especially the most trivial, i.e. those > lacking filters against large data sets), pretty much whack me left and > right > with CPU wedged tools and (completely unacceptable) long waits for results. > > Any suggestions on how to use amazonica in a way where the output is ... > different, and minimal/workable? > > Or am I left with going to another package or writing my own java sdk > api's directly? > > I'm pretty sure the results need to be structures whose relationship to > data values is implicit (and not explicit in map keys). I don't see any > options with amazonica to change this however. > > Thanks for suggestions, forgive me if I've missed something obvious. I'm > just trying to see what's out there and at the same time move along quickly > enough that I can get some usable tools for work (so I can lose all my > python and bash scripts for various interfaces, I want clojure!). > > - Dave > > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.