I've struggled with some huge object count transfers and went back and forth between aws s3 cli and s3cmd.
aws s3 cli seems to edge s3cmd out on speed and RAM consumption when doing huge object count transfers. However, aws s3 cli seems to choke on large object counts and s3cmd offers so much more in terms of feedback and options I hate to ever resort to aws s3 cli. I encountered a bug with aws s3 cli in that it ceased to sync some dirs fully for reasons yet unknown to me. From aws dev forums posts, it appears others have the same problem with aws s3 sync crapping out. It has something to do with high object counts and/or series of subdirs with similar names. I'm going to revisit that when I have time. Matt having modified the s3cmd --no-check-md5 command to not generate md5 on source files (and thus provide a speed boost) helped me be able to switch back. I increased the RAM in my s3cmd migration hosts and went back to s3cmd. Based on my misperception that s3cmd wrote the source object's mod date to the s3 mod date, to get s3cmd to mimic aws s3 cli (and Amazon Import), I turned on s3cmd --no-preserve. Neither aws s3 cli nor Import maintain source object "creation date" on the s3 side, where by default I thought s3cmd did (I was wrong). Further tests indicate it doesn't either. Since none of these tools can do this, it must be an Amazon side issue preventing it? It is perplexing to me why that option is unavailable as it seems like common practice when using rsync to do data migrations. I know this is not apple to apples, but my comparison point (as I expect is not uncommon) is years of using rsync. At any rate, my s3cmd runs (even with 16 GB of RAM in my hosts) are occasionally terminated by the kernel for RAM over-consumption. I have to not be greedy on source dir selections. :) I expect my use is an edge case. All this said, s3cmd does a great job. Thank you to those of you who develop and maintain it. Mike -- wag...@wagnerone.com "The advantage of a bad memory is that one enjoys several times the same good things for the first time."-Nietzsche ------------------------------------------------------------------------------ Learn Graph Databases - Download FREE O'Reilly Book "Graph Databases" is the definitive new guide to graph databases and their applications. Written by three acclaimed leaders in the field, this first edition is now available. Download your free book today! http://p.sf.net/sfu/NeoTech _______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
