example s3cmd for the below /usr/bin/s3cmd -c /home/ec2-user/.s3cfg sync --delete-removed --no-preserve --verbose --progress /localdirectory/ s3://mybucket/directory/
syncs were running build commit d52d5edcc916512e979917f04abcea19d3a25af7 Date: Sat Apr 12 20:40:16 2014 -0500 s3cmd, when syncing a particular directory to S3 is telling me this: INFO: Found 1567064 local files, 1566681 remote files INFO: Verifying attributes... [list of a few "disabled md5 check" objects] INFO: Summary: 14263 local files to upload, 417 files to remote copy, 0 remote files to delete WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. /file.jpg -> s3://mybucket/file.jpg [1 of 111] ^M 4096 of 83840 4% in 0s 403.67 kB/s^M 12288 of 83840 14% in 0s 1188.59 kB/s failed WARNING: Upload failed: /file.jpg (timed out) WARNING: Retrying on lower speed (throttle=0.00) WARNING: Waiting 3 sec... /file.jpg -> s3://mybucket/file.jpg [1 of 111] and continues to the sync finish... ^M 4096 of 15216 26% in 0s 433.79 kB/s^M 15216 of 15216 100% in 0s 201.74 kB/s done /file.txt -> s3://mybucket/file.txt [14263 of 111] then it begins to do remote copies as applicable for the sync. I've not seen this before where the items to be uploaded in the progress output doesn't match the INFO line statement of objects to upload. I also think I have not encountered the "(timed out)" error before and wonder if maybe something related to that function is somehow mucking up the "objects to be uploaded" count display. Just an idea. Another one from the same set of batched sync commands: INFO: Found 615171 local files, 587979 remote files INFO: Verifying attributes... [list of a few "disabled md5 check" objects] INFO: Summary: 6446 local files to upload, 21557 files to remote copy, 0 remote files to delete WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. /file.png -> s3://mybucket.png [1 of 5741] ... /file.mp3 -> s3://mybucket/file.mp3 [6446 of 5741] ... Done. Uploaded 233380819 bytes in 2578.0 seconds, 88.41 kB/s. Copied 21557 files saving 373663253 bytes transfer. That one also had an initial file timeout... Another one... almost matches! :) INFO: Summary: 68371 local files to upload, 29128 files to remote copy, 0 remote files to delete WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. /file.pdf -> s3://mybucket/file.pdf [1 of 68321] ... completed with "[68371 of 68321]" FWIW, as I inspect the logs, I did see this on subsequent syncs that did not time out on the initial object upload. Here is one that is starting to maybe indicate what's happening maybe... INFO: Found 206744 local files, 206740 remote files INFO: Verifying attributes... INFO: Summary: 6164 local files to upload, 15 files to remote copy, 0 remote files to delete WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. /file.jpg -> s3://mybucket/file.jpg [1 of 4] It's wanting to upload 6164 local files (not sure why though), yet that "[1 of 4]" seems to make more sense with the object count difference (4). Also, some of the syncs ran normal - as expected with equal counts on either side, etc. Thanks, Mike ------------------------------------------------------------------------------ Start Your Social Network Today - Download eXo Platform Build your Enterprise Intranet with eXo Platform Software Java Based Open Source Intranet - Social, Extensible, Cloud Ready Get Started Now And Turn Your Intranet Into A Collaboration Platform http://p.sf.net/sfu/ExoPlatform _______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
