[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Lokal_Profil lokal_pro...@hotmail.com changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=67477 -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Bug 48205 depends on bug 55541, which changed state. Bug 55541 Summary: Segfault when getting EXIF data of some tif images https://bugzilla.wikimedia.org/show_bug.cgi?id=55541 What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Lokal_Profil lokal_pro...@hotmail.com changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #32 from Lokal_Profil lokal_pro...@hotmail.com --- (In reply to comment #30) Uploading them manually is going to have the same problem; the issue lies with extracting stuff EXIF data, so no matter how you try, it's always going to fail. The fix being to patch the library as Tim suggested, or push it through something like imagemagick to see if it normalises/fixes the error the PHP EXIF code doesn't like. If the list above is complete then I'll run the images through imagemagick and see if I can upload them afterwards. Once again many thanks for the help. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #33 from Sam Reed (reedy) s...@reedyboy.net --- Complete lits: reedy@terbium:~$ reedy@terbium:~$ screen -r -rwxrwxrwx 1 root root 66988870 Sep 13 03:21 Värja,_defekt,_Frankrike,_1600-talets_mitt,_klinga_och_fäste_ej_ursprungligen_samhörande_-_Livrustkammaren_-_18215.tif -rwxrwxrwx 1 root root 2482 Sep 20 07:57 Värja,_defekt,_Frankrike,_1600-talets_mitt,_klinga_och_fäste_ej_ursprungligen_samhörande_-_Livrustkammaren_-_18215.txt m_dig-notuploaded: total 1731476 drwxrwxrwx 1 root root 20480 Oct 14 15:13 . drwxrwxrwx 1 root root 4096 Oct 18 17:00 .. -rwxrwxrwx 1 root root 99737948 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_51015.tif -rwxrwxrwx 1 root root 736 Aug 26 14:58 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_51015.txt -rwxrwxrwx 1 root root 64282362 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59574.tif -rwxrwxrwx 1 root root 736 Aug 26 14:57 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59574.txt -rwxrwxrwx 1 root root 55393232 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59575.tif -rwxrwxrwx 1 root root 736 Aug 26 14:58 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59575.txt -rwxrwxrwx 1 root root 87114228 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_68136.tif -rwxrwxrwx 1 root root 736 Aug 26 14:57 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_68136.txt -rwxrwxrwx 1 root root 99750768 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33898.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33898.txt -rwxrwxrwx 1 root root 71847212 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33899.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33899.txt -rwxrwxrwx 1 root root 85470850 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33900.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33900.txt -rwxrwxrwx 1 root root 99762232 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42456.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42456.txt -rwxrwxrwx 1 root root 99748964 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42457.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42457.txt -rwxrwxrwx 1 root root 99753286 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42458.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42458.txt -rwxrwxrwx 1 root root 91795152 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42459.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42459.txt -rwxrwxrwx 1 root root 99763146 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_51016.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_51016.txt -rwxrwxrwx 1 root root 99754792 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_85258.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_85258.txt -rwxrwxrwx 1 root root 126227546 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87778.tif -rwxrwxrwx 1 root root 1815 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87778.txt -rwxrwxrwx 1 root root 126244040 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87779.tif -rwxrwxrwx 1 root root 1699 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87779.txt -rwxrwxrwx 1 root root 126220272 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87780.tif -rwxrwxrwx 1 root root 1537 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87780.txt -rwxrwxrwx 1 root root 126201190 Oct 9 23:01 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87785.tif -rwxrwxrwx 1 root root 1093 Oct 9 23:01 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87785.txt -rwxrwxrwx 1 root root 113828698 Nov 6 2012
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #28 from Sam Reed (reedy) s...@reedyboy.net --- Ok, so these are all done now, bar 26 that fail due to bug 55541 (In reply to comment #13) (In reply to comment #12) Tim, do you know which are the erroneous values that the code is falling over? Just wondering if there's anything we can do to modify those values so we can import the images into Commons Maybe a null conversion in ImageMagick would fix it. Or something like http://paste.tstarling.com/p/IKYsdr.html would probably work as a temporary patch. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #29 from Lokal_Profil lokal_pro...@hotmail.com --- Many thanks! I'll tell the museum to push announcements etc. (probably won't get e-mails until Monday). Are those 26 so problematic that they would fail a manual upload as well? If not then I'll just do them by hand. Noticed two things during the upload. First is that the edit comments seemed to get stuck. I.e. everything since File:Rysk mässkåpa - Livrustkammaren - 35643-negative.tif (201310172030) got its edit comment. Not an actual problem but looks a bit weird A few of the files persistently refuse to generate thumbnails, e.g. * https://commons.wikimedia.org/wiki/File:Vas_med_frostblommor_-_Hallwylska_museet_-_87149.tif * https://commons.wikimedia.org/wiki/File:%C3%84lghudskyller,_Sverige_1600-talets_b%C3%B6rjan_-_Livrustkammaren_-_43603.tif Is there an easy way of seeing if this is due to something during the upload or something intrinsic in the files? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #30 from Sam Reed (reedy) s...@reedyboy.net --- If something won't ever thumbnail, check and see if you can view the original (click on the big thumbnail) So for https://commons.wikimedia.org/wiki/File:Vas_med_frostblommor_-_Hallwylska_museet_-_87149.tif is https://upload.wikimedia.org/wikipedia/commons/c/ca/Vas_med_frostblommor_-_Hallwylska_museet_-_87149.tif It's giving a 404. So it needs reuploading. Same for https://commons.wikimedia.org/wiki/File:%C3%84lghudskyller,_Sverige_1600-talets_b%C3%B6rjan_-_Livrustkammaren_-_43603.tif https://upload.wikimedia.org/wikipedia/commons/b/b9/%C3%84lghudskyller%2C_Sverige_1600-talets_b%C3%B6rjan_-_Livrustkammaren_-_43603.tif I have now re-uploaded them. Uploading them manually is going to have the same problem; the issue lies with extracting stuff EXIF data, so no matter how you try, it's always going to fail. The fix being to patch the library as Tim suggested, or push it through something like imagemagick to see if it normalises/fixes the error the PHP EXIF code doesn't like. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205
--- Comment #31 from Sam Reed (reedy) s...@reedyboy.net ---
(In reply to comment #29)
Noticed two things during the upload.
First is that the edit comments seemed to get stuck. I.e. everything since
File:Rysk mässkåpa - Livrustkammaren - 35643-negative.tif (201310172030)
got
its edit comment. Not an actual problem but looks a bit weird
The first image uploads fine with the correct summary, but all following images
use the same edit summary?
(change visibility) 02:51, 18 October 2013 (diff | hist) . . (+926) . . N
File:Begravningsvapen Strängnäs domkyrka - E 15871 - Livrustkammaren -
26663.tif ({{LSH artwork |artist= |title= {{Title|Rysk mässkåpa.}}
|description= {{sv|Rysk mässkåpa, i formen rundskuren, klockformad med en
öppning för huvudet som är något rundad i nacken och rak i kanterna på sidorna.
Av rött turkiskt siden kallat lamp) (current)
(change visibility) 02:51, 18 October 2013 (diff | hist) . . (+772) . . N
File:Begravningsvapen Strängnäs domkyrka - E 15871 - Livrustkammaren -
26663-negative.tif ({{LSH artwork |artist= |title= {{Title|Rysk mässkåpa.}}
|description= {{sv|Rysk mässkåpa, i formen rundskuren, klockformad med en
öppning för huvudet som är något rundad i nacken och rak i kanterna på sidorna.
Av rött turkiskt siden kallat lamp) (current)
Looks like a bug to me.
--
You are receiving this mail because:
You are on the CC list for the bug.
___
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #26 from Lokal_Profil lokal_pro...@hotmail.com --- Many Thanks. Do we know what triggers bug 55541? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #27 from Sam Reed (reedy) s...@reedyboy.net --- (In reply to comment #26) Many Thanks. Do we know what triggers bug 55541? I don't other than it being some value in the EXIF data that the code doesn't account for correctly, but Tim might be able to elaborate and suggest if we can fix the metadata to then upload it. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #25 from Sam Reed (reedy) s...@reedyboy.net --- All top level folders bar m_b have been uploaded, noting the exceptions below. We would seem to be around a third of the files done - 378.4GB uploaded, 836 to go. The following files from m_dig haven't been uploaded due to bug 55541 reedy@terbium:/media/external/keepers$ ls -al m_dig-notuploaded/ total 1731476 drwxrwxrwx 1 root root 20480 Oct 14 15:13 . drwxrwxrwx 1 root root 4096 Oct 14 16:58 .. -rwxrwxrwx 1 root root 99737948 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_51015.tif -rwxrwxrwx 1 root root 736 Aug 26 14:58 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_51015.txt -rwxrwxrwx 1 root root 64282362 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59574.tif -rwxrwxrwx 1 root root 736 Aug 26 14:57 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59574.txt -rwxrwxrwx 1 root root 55393232 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59575.tif -rwxrwxrwx 1 root root 736 Aug 26 14:58 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59575.txt -rwxrwxrwx 1 root root 87114228 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_68136.tif -rwxrwxrwx 1 root root 736 Aug 26 14:57 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_68136.txt -rwxrwxrwx 1 root root 99750768 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33898.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33898.txt -rwxrwxrwx 1 root root 71847212 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33899.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33899.txt -rwxrwxrwx 1 root root 85470850 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33900.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_33900.txt -rwxrwxrwx 1 root root 99762232 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42456.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42456.txt -rwxrwxrwx 1 root root 99748964 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42457.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42457.txt -rwxrwxrwx 1 root root 99753286 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42458.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42458.txt -rwxrwxrwx 1 root root 91795152 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42459.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_42459.txt -rwxrwxrwx 1 root root 99763146 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_51016.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_51016.txt -rwxrwxrwx 1 root root 99754792 Nov 6 2012 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_85258.tif -rwxrwxrwx 1 root root 717 Aug 26 14:58 Dokumentation,_utställningen_Krigsbyte_2007-2008_-_Livrustkammaren_-_85258.txt -rwxrwxrwx 1 root root 126227546 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87778.tif -rwxrwxrwx 1 root root 1815 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87778.txt -rwxrwxrwx 1 root root 126244040 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87779.tif -rwxrwxrwx 1 root root 1699 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87779.txt -rwxrwxrwx 1 root root 126220272 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87780.tif -rwxrwxrwx 1 root root 1537 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87780.txt -rwxrwxrwx 1 root root 126201190 Oct 9 23:01 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87785.tif -rwxrwxrwx 1 root root 1093 Oct 9 23:01 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87785.txt -rwxrwxrwx 1 root root 113828698 Nov 6 2012 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87786.tif -rwxrwxrwx 1 root root 1124 Aug 26 14:58
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #24 from Sam Reed (reedy) s...@reedyboy.net --- Just as an aside for bug 55541, there seems to be a handful of files that upset the PHP EXIF code. As such, I'm just moving them into a different directory out of the way, so we can continue uploading other files. The files currently are: reedy@terbium:/media/external/keepers$ ls -al notuploaded/ total 657060 drwxrwxrwx 1 root root 4096 Oct 14 00:40 . drwxrwxrwx 1 root root 4096 Oct 14 00:33 .. -rwxrwxrwx 1 root root 99737948 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_51015.tif -rwxrwxrwx 1 root root 736 Aug 26 14:58 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_51015.txt -rwxrwxrwx 1 root root 64282362 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59574.tif -rwxrwxrwx 1 root root 736 Aug 26 14:57 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59574.txt -rwxrwxrwx 1 root root 55393232 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59575.tif -rwxrwxrwx 1 root root 736 Aug 26 14:58 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_59575.txt -rwxrwxrwx 1 root root 87114228 Nov 6 2012 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_68136.tif -rwxrwxrwx 1 root root 736 Aug 26 14:57 Dokumentation,_utställningen_Den_oumbärliga_väskan_-_år_2008_-_Livrustkammaren_-_68136.txt -rwxrwxrwx 1 root root 126220272 Nov 6 2012 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87780.tif -rwxrwxrwx 1 root root 1537 Aug 26 14:58 Pressbilder_till_Ars_11_-_Hallwylska_museet_-_87780.txt -rwxrwxrwx 1 root root 126201190 Oct 9 23:01 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87785.tif -rwxrwxrwx 1 root root 1093 Oct 9 23:01 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87785.txt -rwxrwxrwx 1 root root 113828698 Nov 6 2012 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87786.tif -rwxrwxrwx 1 root root 1124 Aug 26 14:58 Pressbilder_till_Ars_11,_reprofotografi_av_bild_från_resealbum_-_Hallwylska_museet_-_87786.txt -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Sam Reed (reedy) s...@reedyboy.net changed: What|Removed |Added Depends on||55541 -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #19 from Lokal_Profil lokal_pro...@hotmail.com --- Thanks for the update. The red-linked categories seem to be the ones at https://commons.wikimedia.org/wiki/Commons:Batch_uploading/LSH/ObjKeywords#class:_HWY-grupp . I've fixed those. I've encountered the thumbnailing issue before. Sometimes the thumbnail is correctly generated one time only to be gone the next one. Not exactly sure what this depends on. Thanks also for the time estimate. Based on this I'd guessed that (bar anything unexpected) the images could be ingested in roughly two weeks? Either way I'll get the museum started on writing their press release and blog posts. Many thanks, André -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #20 from Sam Reed (reedy) s...@reedyboy.net --- As a heads up, Rob has pointed the Wikimedia Communications department at this bug and this donation. There is a reasonably chance there could be a blog post with regards to this rather large donation of files! :) -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #21 from Sam Reed (reedy) s...@reedyboy.net --- Is it possible that numerous files already exist on commons? I noticed there were some other uploads a while ago, but doing the m_a folder I get the following stats: Found: 2140 Added: 988 Skipped: 1152 Just want to make sure there's no issue here I need to check/resolve -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #22 from Lokal_Profil lokal_pro...@hotmail.com --- I removed all of the photos I uploaded during the testing from the drive before sending it so none of them should have caused any skipping. Is it possible to bring up one or two of the skipped filenames and the files on commons which caused them to skip? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #23 from Sam Reed (reedy) s...@reedyboy.net --- Oh, wait. It's presumably me being stupid. Chances of it running through a whole large folder in one go is quite low; with there being many reasons it may fail, it's going to find one of them. Which was what happened overnight, and I started it again. Of course a load of them are going to be skipped on the second run -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #17 from Lokal_Profil lokal_pro...@hotmail.com --- Thanks. Is there a rough estimate for when the server side upload will be done? Asking since the museum needs to prepare their press release. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Sam Reed (reedy) s...@reedyboy.net changed: What|Removed |Added Priority|Lowest |Normal Assignee|lokal_pro...@hotmail.com|s...@reedyboy.net -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #18 from Sam Reed (reedy) s...@reedyboy.net --- (In reply to comment #17) Thanks. Is there a rough estimate for when the server side upload will be done? Asking since the museum needs to prepare their press release. https://commons.wikimedia.org/w/index.php?title=Special%3AListFileslimit=600user=LSHuploadBot There isn't currently. I've imported the m_o and m_t folders (18G or so) today. Noticed one that didn't thumbnail. Poking it further it seems it might have been one of the uploads that failed. Will go and fix that soonish for you. Broken: https://commons.wikimedia.org/wiki/File:M%C3%B6beluppsats,_12_delar,_VI:II:A.a.11_-_Hallwylska_museet_-_39426.tif Relatedly, there's some redlink categories such as https://commons.wikimedia.org/w/index.php?title=Category:Furniture_in_the_Hallwyl_museumaction=editredlink=1 Seems it did ~520 images (18G) in around 100 (97, lets round up) minutes or so... 1.3TB of images = 1331.2G (1331.2/18)*100/60/24 = 5 days at full speed Obviously, we don't want to overload our servers with work (and cause other problems), though the image scalers would only be creating images when anyone tries to view thumbnails of those images - Special:NewFiles etc. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #15 from Rob Halsell rhals...@wikimedia.org --- Please note that Lokal has requested the following for the image import: When it is time the uploads should be tied to the LSHuploadBot account rather than my private account since I will pass this account on to the museum for future (much smaller) uploads. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Chris Johnson cmjohn...@wikimedia.org changed: What|Removed |Added CC||cmjohn...@wikimedia.org --- Comment #16 from Chris Johnson cmjohn...@wikimedia.org --- (In reply to comment #14) Drive has now been sent (via FedEx) The drive arrived to the Virginia data center today. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #14 from Lokal_Profil lokal_pro...@hotmail.com --- Drive has now been sent (via FedEx) -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #11 from Lokal_Profil lokal_pro...@hotmail.com --- Drive is now ready! If you supply me with address/delivery instructions I'll post it sometime next week. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #13 from Rob Halsell rhals...@wikimedia.org --- I've emailed Lokal directly, including both office and datacenter addresses. If the shipment is via USPS, it will come to the office first, so I can send it on via FedEx. (Datacenter won't receive USPS for customers.) -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #12 from Rob Halsell rhals...@wikimedia.org --- So we can have the drive shipped directly to our datacenter, but there are a few restrictions for it: - Shipment cannot go USPS, has to go via FedEx, UPS, or some other third party vendor. - We have to put a ticket in with the datacenter, so we'll need the tracking information. I'm not entirely comfortable just putting our address for the datacenter in the ticket. So I'll be emailing the requestor directly with the address. It is not exactly secret, but may as well not paste it here. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #10 from Lokal_Profil lokal_pro...@hotmail.com --- Ok. As suggested the naming scheme is ImageFileName.ext and ImageFileName.txt, where ImageFileName is the target filename on Commons. After more holiday related delays the drive is now almost ready to be shipped. Barring any complaints on Commons to the latest file descriptions I have only to fix a few files which weren't converted correctly by Imagemagick and make a backup of everything to a separate drive. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Sam Reed (reedy) s...@reedyboy.net changed: What|Removed |Added CC||s...@reedyboy.net --- Comment #9 from Sam Reed (reedy) s...@reedyboy.net --- (In reply to comment #8) A few practical questions. Currently the files are placed in several sub-directories, I assume this won't be a problem (all files on the drive are intended for upload). Filename-wise, should each file have the same filename as the target on Commons or is it ok to have the target filename be the first line in the description file? If either is an issue then I'll rename and shift them all into the same directory prior to sending the drive. For automation purposes having ImageFileName.ext and ImageFileName.txt or similar would be the easiest way, with ImageFileName being where you want it to go on Commons. The usually used import script can search and import recursively. So having them in whatever sort of structure should be fine to make organisation on your end easier, and have no difference on the final upload. That is, as long as the descriptions files are in the same directory as the image file. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #8 from Lokal_Profil lokal_pro...@hotmail.com --- (In reply to comment #7) (In reply to comment #4) I believe that we can do this. Sam and I discussed, and this seems like a perfect case for Resolved/Later (if we still had that resolution), so I'm reassigning to Andre to figure out the right resolution for this. This should probably be assigned to the reporter (Lokal_Profil) until the drive has been sent, and after it has been sent, reassigned to somebody in EQIAD/ops. Lokal_Profil: Any progress / status update that you could share? :) Sorry for the silence. Have the drive and have built the scripts for outputting the descriptions. Awaiting some additional mapping between keywords and categories and will then proceed with a few test uploads (200-500 depending on feedback). Due to summer vacations (most people are away in July) the drive probably won't be ready to ship until end of August though. A few practical questions. Currently the files are placed in several sub-directories, I assume this won't be a problem (all files on the drive are intended for upload). Filename-wise, should each file have the same filename as the target on Commons or is it ok to have the target filename be the first line in the description file? If either is an issue then I'll rename and shift them all into the same directory prior to sending the drive. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Assignee|aklap...@wikimedia.org |lokal_pro...@hotmail.com --- Comment #7 from Andre Klapper aklap...@wikimedia.org --- (In reply to comment #4) I believe that we can do this. Sam and I discussed, and this seems like a perfect case for Resolved/Later (if we still had that resolution), so I'm reassigning to Andre to figure out the right resolution for this. This should probably be assigned to the reporter (Lokal_Profil) until the drive has been sent, and after it has been sent, reassigned to somebody in EQIAD/ops. Lokal_Profil: Any progress / status update that you could share? :) -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #5 from Lokal_Profil lokal_pro...@hotmail.com --- Many thanks. I'll drop a new comment here when we have fixed the descriptions and gotten feedback from the first test uploads. Would the drive have to be sent to the data centre in the US or does the one in Amsterdam have the same functionality/possibility? -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Sam Reed (reedy) s...@reedyboy.net changed: What|Removed |Added Priority|Normal |Lowest --- Comment #6 from Sam Reed (reedy) s...@reedyboy.net --- (In reply to comment #5) Would the drive have to be sent to the data centre in the US or does the one in Amsterdam have the same functionality/possibility? It would have to be in the US (EQIAD - Washington DC), the Amsterdam one isn't really setup for working on these sorts of requests, and would be quicker to get processed in EQIAD anyway -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Rob Lanphier ro...@wikimedia.org changed: What|Removed |Added Priority|Lowest |Normal CC||fai...@wikimedia.org, ||ro...@wikimedia.org Assignee|s...@reedyboy.net|aklap...@wikimedia.org --- Comment #4 from Rob Lanphier ro...@wikimedia.org --- I believe that we can do this. Sam and I discussed, and this seems like a perfect case for Resolved/Later (if we still had that resolution), so I'm reassigning to Andre to figure out the right resolution for this. I'm cc'ing Faidon so that he's aware of this, and can weigh in if needed. -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Assignee|wikibugs-l@lists.wikimedia. |s...@reedyboy.net |org | --- Comment #2 from Andre Klapper aklap...@wikimedia.org --- Tentatively assigning this to Reedy - feel free to reassign. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 --- Comment #3 from Sam Reed (reedy) s...@reedyboy.net --- Yeah, this has been done before, so there is a likelihood of being able to do so again. If you want to leave a message on this bug when you're in a position to actually send the hard drive, we can loop in the relevant people to move forward, and should be able to accommodate. If we give you the address for the data centre, then after sending it by an appropriate tracked courier, this should be sufficient. We need to follow procedures to inform the data centre of pending deliveries etc. But this won't be a problem -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 48205] Server-side upload of large number of files to Commons
https://bugzilla.wikimedia.org/show_bug.cgi?id=48205 Tomasz W. Kozlowski tom...@twkozlowski.net changed: What|Removed |Added Priority|Unprioritized |Lowest CC||roan.katt...@gmail.com --- Comment #1 from Tomasz W. Kozlowski tom...@twkozlowski.net --- CC'ing Roan and Sam since they are some of the people responsible for server-side uploads. (Roan uploaded Wikimania 2011 videos to Commons [lots of data!], and Sam deals with upload requests on a regular basis.) -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
