Actually, let me withdraw the question for now. If I call an unfiltered
(describe-images) on my account I'll get ~27,900 images. It takes 70
seconds to retrieve them using the Java api
(from clojure).
If I then print (str image) for all those images to a file, that makes adds
another 153 seconds for a total of 223 seconds. Presumably that's the
normal java toString() method invocation.
If I print out the Amazonica version of it, it takes 195 seconds,
presumably because we're sharing keyword references internally and so
abusing memory less overall (just a wild guess).
So if I do the native calls and cherry pick the information I want (like
the java EC2 CLI does), then I can get the time down significantly.
Otherwise Amazonica is probably doing a reasonable job given what I'm
asking of it.
And, in the wisdom gained department, never do unfiltered (describe-images)
requests if you can help it :-)
On Fri, Mar 28, 2014 at 12:52 PM, Michael Cohen <mcohe...@gmail.com> wrote:
> time ec2-describe-images -a > ec2-cli-images.txt
>
> real 1m26.401s
> user 0m6.551s
> sys 0m1.159s
>
> and writes a 7.5MB file to disk. Note the -a flag, to list all of the
> available public images.
>
> in a repl,
>
> (time (spit "clj-awz-images.txt" (describe-images)))
>
> "Elapsed time: 90258.47 msecs"
>
> and writes an 18MB file to disk containing all the available public
> images.
>
> Am I missing something?
>
> You can also pass a list of filters to the call to narrow the result.
>
>
>
> On Friday, March 28, 2014 7:59:48 AM UTC-7, Dave Tenny wrote:
>>
>> I'm trying to code some amazonica based solutions in a nontrivial AWS
>> environment.
>> I work with many AWS accounts and it isn't unusual to see a thousand
>> instances running on one account, and similar excesses in other types of
>> AWS resources. So if you're going an ec2-describe-instances (or amazonica
>> equivalent), it needs not to choke in this environment.
>>
>> I like the way amazonica does all the bean marshalling for me so I can
>> express queries simply. But the returned datasets need to be more
>> pragmatic/performant.
>>
>> The problem for me is that Amazonica doesn't seem up to the task of
>> dealing with queries that return large volumes of data.
>> It has nothing to do with reflection I suspect, and more to do with
>> unwieldy amounts of duplicate information in the result unmarshalling
>> process.
>> The "clojure all the way down" philosophy results of duplicated
>> information and just printing the result to a file takes a long time.
>> If I accidentally let the output go to an emacs cider repl buffer, then
>> things get so wedged up to the point I may as well kill -9 emacs.
>> (Known cider repl issues here, it isn't all amazonica).
>>
>> For example: here's how long it takes to run the java based ec2 cli to
>> describe instances on an account:
>>
>> $ time ec2-describe-images >/tmp/ec2-cli-images.out
>>
>> real 0m11.484s
>> user 0m2.564s
>> sys 0m0.129s
>>
>>
>> And here's how long it takes from a 'lein repl' to run the same query on
>> the same account:
>>
>> (time (with-output ["/tmp/clj-awz-images.out"] (println
>> (ec2/describe-images))))
>> "Elapsed time: 194685.552683 msecs"
>>
>> Now the amount of data being printed by the EC2 CLI is of course much
>> different than the output from Amazonica,
>> amazonica is returning everything in gory duplicate map detail, ec2 is
>> not, as evidenced by the relative output sizes:
>>
>> -rw-rw-r--. 1 dave dave 17201290 Mar 28 10:35 clj-awz-images.out
>> -rw-rw-r--. 1 dave dave 99342 Mar 28 10:26 ec2-cli-images.out.11.5s
>>
>> Where the amazonica output starts with:
>> {:images [{:hypervisor xen, :state available, :virtualization-type
>> paravirtual, :root-device-type instance-store,
>> ... and goes on like that with duplicate keywords all the way down.
>>
>> Anyway, my goal isn't to turn amazonica into ec2 cli. But even the most
>> trivial operations in amazonica (especially the most trivial, i.e. those
>> lacking filters against large data sets), pretty much whack me left and
>> right
>> with CPU wedged tools and (completely unacceptable) long waits for
>> results.
>>
>> Any suggestions on how to use amazonica in a way where the output is ...
>> different, and minimal/workable?
>>
>> Or am I left with going to another package or writing my own java sdk
>> api's directly?
>>
>> I'm pretty sure the results need to be structures whose relationship to
>> data values is implicit (and not explicit in map keys). I don't see any
>> options with amazonica to change this however.
>>
>> Thanks for suggestions, forgive me if I've missed something obvious. I'm
>> just trying to see what's out there and at the same time move along quickly
>> enough that I can get some usable tools for work (so I can lose all my
>> python and bash scripts for various interfaces, I want clojure!).
>>
>> - Dave
>>
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/QCBM_el5_78/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
