| eflyjason updated the task description. (Show Details) |
CHANGES TO TASK DESCRIPTION
Consider a scenario where on a machine, a bot that processes the dump starts starts right after the download script starts, and that the dump processor is faster andthan the downloader. In the current implementation, the processor will eventually find the dump incomplete, and the behavior of the processor will be often undefined (eg: crash).
One implementation is to perform the download on a temporary file. In the wild '<filename>.part' is often used. Move the temporary file over tto the target file ifff the download is complete and verified, or delete the temporary file if it fails. In *nix implementation, the currently running processors will not be disrupted as the file descriptors of their opened dump file will continue to point to the old moved-over now-nonexistent file, and will happily read it without ever knowing a new version has arrived, till the dump file is reopened for another run.
...
1. Make sure the target file is //always// a complete file; only commit iff the file is completely downloaded (and verified with checksum), or discard if anything goes wrong.Consider a scenario where on a machine, a bot that processes the dump starts starts right after the download script starts, and that the dump processor is faster andthan the downloader. In the current implementation, the processor will eventually find the dump incomplete, and the behavior of the processor will be often undefined (eg: crash).
One implementation is to perform the download on a temporary file. In the wild '<filename>.part' is often used. Move the temporary file over tto the target file ifff the download is complete and verified, or delete the temporary file if it fails. In *nix implementation, the currently running processors will not be disrupted as the file descriptors of their opened dump file will continue to point to the old moved-over now-nonexistent file, and will happily read it without ever knowing a new version has arrived, till the dump file is reopened for another run.
...
TASK DETAIL
EMAIL PREFERENCES
To: eflyjason
Cc: Framawiki, Aklapper, Xqt, jayvdb, siebrand, Zoranzoki21, eflyjason, pywikibot-bugs-list, zhuyifei1999, Bright1055, Toppole69, Mine0901, Jayprakash12345, Magul, Tbscho, MayS, Beeyan, Mdupont, JJMC89, MtDu, D3r1ck01, Avicennasis, Dalba, Masti, Alchimista, Rxy
Cc: Framawiki, Aklapper, Xqt, jayvdb, siebrand, Zoranzoki21, eflyjason, pywikibot-bugs-list, zhuyifei1999, Bright1055, Toppole69, Mine0901, Jayprakash12345, Magul, Tbscho, MayS, Beeyan, Mdupont, JJMC89, MtDu, D3r1ck01, Avicennasis, Dalba, Masti, Alchimista, Rxy
_______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
