Re: [CMake] file( DOWNLOAD ) problem
I have some weird logic for the download. If the file is NOT on the
server, that is not an error -- it simply means we do not have
includes, or do not have binaries (either is not erroneous, as not
every library will have or need both). So I've had to write custom
logic for each protocol to handle this (doesn't seem safe). Is there a
better way to verify if a file exists or not on an FTP/HTTP server? If
not, could I possibly add a feature for this? Such as:
file( EXISTS path result )
(This would be slightly more functional than if(EXISTS), as it will
work with FTP/HTTP/HTTPS as well as file paths)
Here is the logic I use currently (as I said, it seems unreliable, I
had to replicate a missing file on each protocol to see what error it
returned). What's most concerning is that I think all errors return
error code 22 on HTTP, so I have no way of knowing reliably if a file
exists or not:
file( DOWNLOAD ${source} ${destination} STATUS status )
list( GET status 0 error_code )
if( error_code )
file( REMOVE ${destination} )
# If the file cannot be found on the server,
# we assume it isn't supposed to have it.
# We fail with all other errors.
if( NOT error_code EQUAL 19 AND # FTP: Invalid file / file not found
NOT error_code EQUAL 22 # HTTP: HTTP response code
said error
)
list( GET status 1 error_message )
message( FATAL_ERROR Failed to download file via
HTTP/FTP: ${error_message} )
endif()
endif()
On Fri, Sep 28, 2012 at 7:50 PM, David Cole david.c...@kitware.com wrote:
What I'm saying is that it _does_ work properly, which is the problem
Ha!!
Wish all our problems were that things worked properly!
:-)
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
CMake downloads our third party libraries from a central repository and we have a manifest.cmake module where we define the following: - Library alias (the library's base name, such as boost, bdb, openssl) - Library version (e.g. 2.1.5) - Library iteration (A counter that is incremented if a library changes remotely without version # increasing (such as if we rebuild the same version of the library and it must be re-served)) My third party download logic knows to download the following files: repo/alias/version/include.7z repo/alias/version/platform.7z In this case, platform will represent the toolchain -- such as vc9sp1.7z for the lib bin files for visual studio 2008 SP1. I have 2 files here, so I'd need 2 MD5 values recorded in my manifest somewhere, but since I have 1 line per library (not per file that will be downloaded) it wouldn't work out very well. I want to keep my manifest simple and easy to look at and modify, adding a bunch of MD5 values will make it messy and harder to upgrade libraries (right now I just drop files on a server and add or modify a line in the manifest. Having MD5s would mean I would have to run another tool to calculate the MD5 and then stuff it somewhere in the manifest module) If you have some ideas on how to make this fit well into my system I'm all for that, but I guess if not then I'll have to rely on assumptions :( However I strongly believe that CMake's file DOWNLOAD should do more checks to make sure that the data received is valid. I will look at the code later to see if there is more that can be done. On Wed, Sep 26, 2012 at 11:20 PM, David Cole david.c...@kitware.com wrote: On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey rcdailey.li...@gmail.com wrote: To do MD5 checks, I need to somehow record the expected MD5 somewhere, which isn't very maintainable. I provide a list of third party libraries that CMake should download from a central third party repository here at work. It is a trusted source, because we know it is, so we don't need to verify the MD5. However, if I could request the MD5 first, and then download, then compare the MD5 the server gave me with what I actually downloaded, that would certainly work just to verify the complete file was downloaded. Other than that, I'll have to rely on the status of the operation... but I don't like that the destination file is created prior to any writes being possible by CMake (it can't write anything if no data was received, so why doesn't it create the file once it has a write buffer?) Recording the MD5 somewhere is the only way to have a reasonable re-assurance that what you've asked for is what you're getting from a network operation. It seems to me that it could be made maintainable if you centralize the knowledge of the checksums in a file that is changed whenever any of the downloadable files is changed. I guess we figure it's no use downloading bits over the network if you can't even open a (presumably local) output file for writing... so we try to open the output file for writing first, and if it succeeds, then we start grabbing bits from the network and writing them into the file as we receive them. There is room for improvement in the file(DOWNLOAD implementation, but it is the way it is right now (and will be for 2.8.10 as well...) Start proposing improvements for it now, and submitting patches to make stuff better for 2.8.11 and/or beyond. :-) HTH, David -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
On Fri, Sep 28, 2012 at 3:30 PM, Robert Dailey rcdailey.li...@gmail.comwrote: CMake downloads our third party libraries from a central repository and we have a manifest.cmake module where we define the following: - Library alias (the library's base name, such as boost, bdb, openssl) - Library version (e.g. 2.1.5) - Library iteration (A counter that is incremented if a library changes remotely without version # increasing (such as if we rebuild the same version of the library and it must be re-served)) My third party download logic knows to download the following files: repo/alias/version/include.7z repo/alias/version/platform.7z In this case, platform will represent the toolchain -- such as vc9sp1.7z for the lib bin files for visual studio 2008 SP1. I have 2 files here, so I'd need 2 MD5 values recorded in my manifest somewhere, but since I have 1 line per library (not per file that will be downloaded) it wouldn't work out very well. I want to keep my manifest simple and easy to look at and modify, adding a bunch of MD5 values will make it messy and harder to upgrade libraries (right now I just drop files on a server and add or modify a line in the manifest. Having MD5s would mean I would have to run another tool to calculate the MD5 and then stuff it somewhere in the manifest module) If you have some ideas on how to make this fit well into my system I'm all for that, but I guess if not then I'll have to rely on assumptions :( However I strongly believe that CMake's file DOWNLOAD should do more checks to make sure that the data received is valid. I will look at the code later to see if there is more that can be done. On Wed, Sep 26, 2012 at 11:20 PM, David Cole david.c...@kitware.com wrote: On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey rcdailey.li...@gmail.com wrote: To do MD5 checks, I need to somehow record the expected MD5 somewhere, which isn't very maintainable. I provide a list of third party libraries that CMake should download from a central third party repository here at work. It is a trusted source, because we know it is, so we don't need to verify the MD5. However, if I could request the MD5 first, and then download, then compare the MD5 the server gave me with what I actually downloaded, that would certainly work just to verify the complete file was downloaded. Other than that, I'll have to rely on the status of the operation... but I don't like that the destination file is created prior to any writes being possible by CMake (it can't write anything if no data was received, so why doesn't it create the file once it has a write buffer?) Recording the MD5 somewhere is the only way to have a reasonable re-assurance that what you've asked for is what you're getting from a network operation. It seems to me that it could be made maintainable if you centralize the knowledge of the checksums in a file that is changed whenever any of the downloadable files is changed. I guess we figure it's no use downloading bits over the network if you can't even open a (presumably local) output file for writing... so we try to open the output file for writing first, and if it succeeds, then we start grabbing bits from the network and writing them into the file as we receive them. There is room for improvement in the file(DOWNLOAD implementation, but it is the way it is right now (and will be for 2.8.10 as well...) Start proposing improvements for it now, and submitting patches to make stuff better for 2.8.11 and/or beyond. :-) HTH, David You can rely on the STATUS to see if there were any errors during the download. If the error code is 0, then you got whatever was on the server. You can rely on that. So, if you don't want to use a hash, you can rely on STATUS. I do not know of any case that reports a 0 status code, but gives an incorrect file result. What you *can't* rely on is that the correct thing was on the server. And to validate that, you should use checksums of some sort. (If you can't or don't want to, that's fine. To each his own.) Starting with CMake 2.8.10, there will be EXPECTED_HASH and you can use the hashing algorithm of your choice rather than just the MD5 that we had in 2.8.9 and earlier... Also new in 2.8.10, the Kitware provided pre-built binaries will link to OpenSSL such that we can handle downloading files from https://; URLs. HTH, David -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
On Fri, Sep 28, 2012 at 2:58 PM, David Cole david.c...@kitware.com wrote: On Fri, Sep 28, 2012 at 3:30 PM, Robert Dailey rcdailey.li...@gmail.com wrote: CMake downloads our third party libraries from a central repository and we have a manifest.cmake module where we define the following: - Library alias (the library's base name, such as boost, bdb, openssl) - Library version (e.g. 2.1.5) - Library iteration (A counter that is incremented if a library changes remotely without version # increasing (such as if we rebuild the same version of the library and it must be re-served)) My third party download logic knows to download the following files: repo/alias/version/include.7z repo/alias/version/platform.7z In this case, platform will represent the toolchain -- such as vc9sp1.7z for the lib bin files for visual studio 2008 SP1. I have 2 files here, so I'd need 2 MD5 values recorded in my manifest somewhere, but since I have 1 line per library (not per file that will be downloaded) it wouldn't work out very well. I want to keep my manifest simple and easy to look at and modify, adding a bunch of MD5 values will make it messy and harder to upgrade libraries (right now I just drop files on a server and add or modify a line in the manifest. Having MD5s would mean I would have to run another tool to calculate the MD5 and then stuff it somewhere in the manifest module) If you have some ideas on how to make this fit well into my system I'm all for that, but I guess if not then I'll have to rely on assumptions :( However I strongly believe that CMake's file DOWNLOAD should do more checks to make sure that the data received is valid. I will look at the code later to see if there is more that can be done. On Wed, Sep 26, 2012 at 11:20 PM, David Cole david.c...@kitware.com wrote: On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey rcdailey.li...@gmail.com wrote: To do MD5 checks, I need to somehow record the expected MD5 somewhere, which isn't very maintainable. I provide a list of third party libraries that CMake should download from a central third party repository here at work. It is a trusted source, because we know it is, so we don't need to verify the MD5. However, if I could request the MD5 first, and then download, then compare the MD5 the server gave me with what I actually downloaded, that would certainly work just to verify the complete file was downloaded. Other than that, I'll have to rely on the status of the operation... but I don't like that the destination file is created prior to any writes being possible by CMake (it can't write anything if no data was received, so why doesn't it create the file once it has a write buffer?) Recording the MD5 somewhere is the only way to have a reasonable re-assurance that what you've asked for is what you're getting from a network operation. It seems to me that it could be made maintainable if you centralize the knowledge of the checksums in a file that is changed whenever any of the downloadable files is changed. I guess we figure it's no use downloading bits over the network if you can't even open a (presumably local) output file for writing... so we try to open the output file for writing first, and if it succeeds, then we start grabbing bits from the network and writing them into the file as we receive them. There is room for improvement in the file(DOWNLOAD implementation, but it is the way it is right now (and will be for 2.8.10 as well...) Start proposing improvements for it now, and submitting patches to make stuff better for 2.8.11 and/or beyond. :-) HTH, David You can rely on the STATUS to see if there were any errors during the download. If the error code is 0, then you got whatever was on the server. You can rely on that. So, if you don't want to use a hash, you can rely on STATUS. I do not know of any case that reports a 0 status code, but gives an incorrect file result. What you *can't* rely on is that the correct thing was on the server. And to validate that, you should use checksums of some sort. (If you can't or don't want to, that's fine. To each his own.) Starting with CMake 2.8.10, there will be EXPECTED_HASH and you can use the hashing algorithm of your choice rather than just the MD5 that we had in 2.8.9 and earlier... Also new in 2.8.10, the Kitware provided pre-built binaries will link to OpenSSL such that we can handle downloading files from https://; URLs. In my tests, I've found that redirects can affect the return code of STATUS. For example, if I try to initiate a download of a file that doesn't really exist, the HTTP server may return a dummy file, in that case it would be downloaded just fine no matter what the URL or filename is, and status wouldn't know the difference. However for FTP URLs, it is generally more honest (since HTTP can do funny things, like lie to you). -- Powered by
Re: [CMake] file( DOWNLOAD ) problem
On Fri, Sep 28, 2012 at 4:42 PM, Robert Dailey rcdailey.li...@gmail.com wrote: On Fri, Sep 28, 2012 at 2:58 PM, David Cole david.c...@kitware.com wrote: On Fri, Sep 28, 2012 at 3:30 PM, Robert Dailey rcdailey.li...@gmail.com wrote: CMake downloads our third party libraries from a central repository and we have a manifest.cmake module where we define the following: - Library alias (the library's base name, such as boost, bdb, openssl) - Library version (e.g. 2.1.5) - Library iteration (A counter that is incremented if a library changes remotely without version # increasing (such as if we rebuild the same version of the library and it must be re-served)) My third party download logic knows to download the following files: repo/alias/version/include.7z repo/alias/version/platform.7z In this case, platform will represent the toolchain -- such as vc9sp1.7z for the lib bin files for visual studio 2008 SP1. I have 2 files here, so I'd need 2 MD5 values recorded in my manifest somewhere, but since I have 1 line per library (not per file that will be downloaded) it wouldn't work out very well. I want to keep my manifest simple and easy to look at and modify, adding a bunch of MD5 values will make it messy and harder to upgrade libraries (right now I just drop files on a server and add or modify a line in the manifest. Having MD5s would mean I would have to run another tool to calculate the MD5 and then stuff it somewhere in the manifest module) If you have some ideas on how to make this fit well into my system I'm all for that, but I guess if not then I'll have to rely on assumptions :( However I strongly believe that CMake's file DOWNLOAD should do more checks to make sure that the data received is valid. I will look at the code later to see if there is more that can be done. On Wed, Sep 26, 2012 at 11:20 PM, David Cole david.c...@kitware.com wrote: On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey rcdailey.li...@gmail.com wrote: To do MD5 checks, I need to somehow record the expected MD5 somewhere, which isn't very maintainable. I provide a list of third party libraries that CMake should download from a central third party repository here at work. It is a trusted source, because we know it is, so we don't need to verify the MD5. However, if I could request the MD5 first, and then download, then compare the MD5 the server gave me with what I actually downloaded, that would certainly work just to verify the complete file was downloaded. Other than that, I'll have to rely on the status of the operation... but I don't like that the destination file is created prior to any writes being possible by CMake (it can't write anything if no data was received, so why doesn't it create the file once it has a write buffer?) Recording the MD5 somewhere is the only way to have a reasonable re-assurance that what you've asked for is what you're getting from a network operation. It seems to me that it could be made maintainable if you centralize the knowledge of the checksums in a file that is changed whenever any of the downloadable files is changed. I guess we figure it's no use downloading bits over the network if you can't even open a (presumably local) output file for writing... so we try to open the output file for writing first, and if it succeeds, then we start grabbing bits from the network and writing them into the file as we receive them. There is room for improvement in the file(DOWNLOAD implementation, but it is the way it is right now (and will be for 2.8.10 as well...) Start proposing improvements for it now, and submitting patches to make stuff better for 2.8.11 and/or beyond. :-) HTH, David You can rely on the STATUS to see if there were any errors during the download. If the error code is 0, then you got whatever was on the server. You can rely on that. So, if you don't want to use a hash, you can rely on STATUS. I do not know of any case that reports a 0 status code, but gives an incorrect file result. What you *can't* rely on is that the correct thing was on the server. And to validate that, you should use checksums of some sort. (If you can't or don't want to, that's fine. To each his own.) Starting with CMake 2.8.10, there will be EXPECTED_HASH and you can use the hashing algorithm of your choice rather than just the MD5 that we had in 2.8.9 and earlier... Also new in 2.8.10, the Kitware provided pre-built binaries will link to OpenSSL such that we can handle downloading files from https://; URLs. In my tests, I've found that redirects can affect the return code of STATUS. For example, if I try to initiate a download of a file that doesn't really exist, the HTTP server may return a dummy file, in that case it would be downloaded just fine no matter what the URL or filename is, and status wouldn't know the difference. However for FTP URLs, it is generally more
Re: [CMake] file( DOWNLOAD ) problem
On Fri, Sep 28, 2012 at 4:42 PM, Robert Dailey rcdailey.li...@gmail.comwrote: On Fri, Sep 28, 2012 at 2:58 PM, David Cole david.c...@kitware.com wrote: On Fri, Sep 28, 2012 at 3:30 PM, Robert Dailey rcdailey.li...@gmail.com wrote: CMake downloads our third party libraries from a central repository and we have a manifest.cmake module where we define the following: - Library alias (the library's base name, such as boost, bdb, openssl) - Library version (e.g. 2.1.5) - Library iteration (A counter that is incremented if a library changes remotely without version # increasing (such as if we rebuild the same version of the library and it must be re-served)) My third party download logic knows to download the following files: repo/alias/version/include.7z repo/alias/version/platform.7z In this case, platform will represent the toolchain -- such as vc9sp1.7z for the lib bin files for visual studio 2008 SP1. I have 2 files here, so I'd need 2 MD5 values recorded in my manifest somewhere, but since I have 1 line per library (not per file that will be downloaded) it wouldn't work out very well. I want to keep my manifest simple and easy to look at and modify, adding a bunch of MD5 values will make it messy and harder to upgrade libraries (right now I just drop files on a server and add or modify a line in the manifest. Having MD5s would mean I would have to run another tool to calculate the MD5 and then stuff it somewhere in the manifest module) If you have some ideas on how to make this fit well into my system I'm all for that, but I guess if not then I'll have to rely on assumptions :( However I strongly believe that CMake's file DOWNLOAD should do more checks to make sure that the data received is valid. I will look at the code later to see if there is more that can be done. On Wed, Sep 26, 2012 at 11:20 PM, David Cole david.c...@kitware.com wrote: On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey rcdailey.li...@gmail.com wrote: To do MD5 checks, I need to somehow record the expected MD5 somewhere, which isn't very maintainable. I provide a list of third party libraries that CMake should download from a central third party repository here at work. It is a trusted source, because we know it is, so we don't need to verify the MD5. However, if I could request the MD5 first, and then download, then compare the MD5 the server gave me with what I actually downloaded, that would certainly work just to verify the complete file was downloaded. Other than that, I'll have to rely on the status of the operation... but I don't like that the destination file is created prior to any writes being possible by CMake (it can't write anything if no data was received, so why doesn't it create the file once it has a write buffer?) Recording the MD5 somewhere is the only way to have a reasonable re-assurance that what you've asked for is what you're getting from a network operation. It seems to me that it could be made maintainable if you centralize the knowledge of the checksums in a file that is changed whenever any of the downloadable files is changed. I guess we figure it's no use downloading bits over the network if you can't even open a (presumably local) output file for writing... so we try to open the output file for writing first, and if it succeeds, then we start grabbing bits from the network and writing them into the file as we receive them. There is room for improvement in the file(DOWNLOAD implementation, but it is the way it is right now (and will be for 2.8.10 as well...) Start proposing improvements for it now, and submitting patches to make stuff better for 2.8.11 and/or beyond. :-) HTH, David You can rely on the STATUS to see if there were any errors during the download. If the error code is 0, then you got whatever was on the server. You can rely on that. So, if you don't want to use a hash, you can rely on STATUS. I do not know of any case that reports a 0 status code, but gives an incorrect file result. What you *can't* rely on is that the correct thing was on the server. And to validate that, you should use checksums of some sort. (If you can't or don't want to, that's fine. To each his own.) Starting with CMake 2.8.10, there will be EXPECTED_HASH and you can use the hashing algorithm of your choice rather than just the MD5 that we had in 2.8.9 and earlier... Also new in 2.8.10, the Kitware provided pre-built binaries will link to OpenSSL such that we can handle downloading files from https://; URLs. In my tests, I've found that redirects can affect the return code of STATUS. For example, if I try to initiate a download of a file that doesn't really exist, the HTTP server may return a dummy file, in that case it would be downloaded just fine no matter what
Re: [CMake] file( DOWNLOAD ) problem
On Fri, Sep 28, 2012 at 4:25 PM, David Cole david.c...@kitware.com wrote: What version of CMake are you using? Server side redirects should be followed properly since this commit: http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=ef491f78218e255339278656bf6dc26073fef264 Which has been in CMake for more than 2 years since 2.8.2... What I'm saying is that it _does_ work properly, which is the problem -- if a file doesn't exist on an HTTP server, and the file name was specified incorrectly in the URL, I expect the request to fail (i.e. NOT redirect) but instead I am served a dummy file due to the redirect. Anyway this only happened for images on public websites during my test, but if it's an HTTP server I'm in control of for the purposes of serving third party libs, I can configure it not to redirect and instead return a response with an error code. -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
What I'm saying is that it _does_ work properly, which is the problem Ha!! Wish all our problems were that things worked properly! :-) -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
With file(DOWNLOAD, you should always use the EXPECTED_MD5 argument (or EXPECTED_HASH with nightly CMake, or 2.8.10 or later) to verify success. HTH, David On Wed, Sep 26, 2012 at 5:40 PM, Robert Dailey rcdailey.li...@gmail.comwrote: I'm using the file( DOWNLOAD ) command to download some files from an FTP server. I notice that if I put a completely invalid URL to a file, the destination file is still created in Windows with a size of 0 bytes. This tricks my logic into thinking the download was successful (since I only check if the destination file exists). Is this a bug or a feature? If it's a feature, what is a reliable way I can determine if the download succeeded? -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
2012/9/27 David Cole david.c...@kitware.com: With file(DOWNLOAD, you should always use the EXPECTED_MD5 argument (or EXPECTED_HASH with nightly CMake, or 2.8.10 or later) to verify success. I bet this is not always possible. Why would you **always** know some hash of the file you are downloading? May be STATUS is usable in this case as well: If STATUS var is specified the status of the operation will be put in var. The status is returned in a list of length 2. The first element is the numeric return value for the operation, and the second element is a string value for the error. A 0 numeric error means no error in the operation -- Erk Le gouvernement représentatif n'est pas la démocratie -- http://www.le-message.org -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
On Wed, Sep 26, 2012 at 6:12 PM, Eric Noulard eric.noul...@gmail.com wrote:
2012/9/27 David Cole david.c...@kitware.com:
With file(DOWNLOAD, you should always use the EXPECTED_MD5 argument (or
EXPECTED_HASH with nightly CMake, or 2.8.10 or later) to verify success.
I bet this is not always possible.
Why would you **always** know some hash of the file you are downloading?
If you don't, then you can't know if it came down correctly or not,
even if STATUS tells you that the download succeeded... So. I would
say if you don't know the expected hash, then you can't trust the
source of the download. I personally would not use file(DOWNLOAD to
grab anything of importance unless I knew what it's expected hash is.
May be STATUS is usable in this case as well:
If STATUS var is specified the status of the operation will be put in
var. The status is returned in a list of length 2. The first element
is the numeric return value for the operation, and the second element
is a string value for the error. A 0 numeric error means no error in
the operation
--
Erk
Le gouvernement représentatif n'est pas la démocratie --
http://www.le-message.org
Yes, you can see how file(DOWNLOAD is used from ExternalProject by
inspecting one of its generated download scripts. Here's an example
for downloading the tarball for the zlib project:
message(STATUS downloading...
src='http://zlib.net/zlib-1.2.7.tar.gz'
dst='/Users/davidcole/Downloads/zlib-1.2.7.tar.gz'
timeout='none')
file(DOWNLOAD
http://zlib.net/zlib-1.2.7.tar.gz;
/Users/davidcole/Downloads/zlib-1.2.7.tar.gz
SHOW_PROGRESS
EXPECTED_MD5;60df6a37c56e7c1366cca812414f7b85
# no TIMEOUT
STATUS status
LOG log)
list(GET status 0 status_code)
list(GET status 1 status_string)
if(NOT status_code EQUAL 0)
message(FATAL_ERROR error: downloading
'http://zlib.net/zlib-1.2.7.tar.gz' failed
status_code: ${status_code}
status_string: ${status_string}
log: ${log}
)
endif()
message(STATUS downloading... done)
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
On Wed, Sep 26, 2012 at 6:28 PM, David Cole david.c...@kitware.com wrote:
On Wed, Sep 26, 2012 at 6:12 PM, Eric Noulard eric.noul...@gmail.com wrote:
2012/9/27 David Cole david.c...@kitware.com:
With file(DOWNLOAD, you should always use the EXPECTED_MD5 argument (or
EXPECTED_HASH with nightly CMake, or 2.8.10 or later) to verify success.
I bet this is not always possible.
Why would you **always** know some hash of the file you are downloading?
If you don't, then you can't know if it came down correctly or not,
even if STATUS tells you that the download succeeded... So. I would
say if you don't know the expected hash, then you can't trust the
source of the download. I personally would not use file(DOWNLOAD to
grab anything of importance unless I knew what it's expected hash is.
May be STATUS is usable in this case as well:
If STATUS var is specified the status of the operation will be put in
var. The status is returned in a list of length 2. The first element
is the numeric return value for the operation, and the second element
is a string value for the error. A 0 numeric error means no error in
the operation
--
Erk
Le gouvernement représentatif n'est pas la démocratie --
http://www.le-message.org
Yes, you can see how file(DOWNLOAD is used from ExternalProject by
inspecting one of its generated download scripts. Here's an example
for downloading the tarball for the zlib project:
message(STATUS downloading...
src='http://zlib.net/zlib-1.2.7.tar.gz'
dst='/Users/davidcole/Downloads/zlib-1.2.7.tar.gz'
timeout='none')
file(DOWNLOAD
http://zlib.net/zlib-1.2.7.tar.gz;
/Users/davidcole/Downloads/zlib-1.2.7.tar.gz
SHOW_PROGRESS
EXPECTED_MD5;60df6a37c56e7c1366cca812414f7b85
# no TIMEOUT
STATUS status
LOG log)
list(GET status 0 status_code)
list(GET status 1 status_string)
if(NOT status_code EQUAL 0)
message(FATAL_ERROR error: downloading
'http://zlib.net/zlib-1.2.7.tar.gz' failed
status_code: ${status_code}
status_string: ${status_string}
log: ${log}
)
endif()
message(STATUS downloading... done)
Plus. if you use the EXPECTED_MD5 argument, then file(DOWNLOAD can
short-circuit/by-pass the network activity and associated wait time if
the file already exists and already has the expected hash value... So
it saves gobs and gobs of time for people executing the same
file(DOWNLOAD command repeatedly.
:-)
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
To do MD5 checks, I need to somehow record the expected MD5 somewhere,
which isn't very maintainable.
I provide a list of third party libraries that CMake should download
from a central third party repository here at work. It is a trusted
source, because we know it is, so we don't need to verify the MD5.
However, if I could request the MD5 first, and then download, then
compare the MD5 the server gave me with what I actually downloaded,
that would certainly work just to verify the complete file was
downloaded.
Other than that, I'll have to rely on the status of the operation...
but I don't like that the destination file is created prior to any
writes being possible by CMake (it can't write anything if no data was
received, so why doesn't it create the file once it has a write
buffer?)
On Wed, Sep 26, 2012 at 5:30 PM, David Cole david.c...@kitware.com wrote:
On Wed, Sep 26, 2012 at 6:28 PM, David Cole david.c...@kitware.com wrote:
On Wed, Sep 26, 2012 at 6:12 PM, Eric Noulard eric.noul...@gmail.com wrote:
2012/9/27 David Cole david.c...@kitware.com:
With file(DOWNLOAD, you should always use the EXPECTED_MD5 argument (or
EXPECTED_HASH with nightly CMake, or 2.8.10 or later) to verify success.
I bet this is not always possible.
Why would you **always** know some hash of the file you are downloading?
If you don't, then you can't know if it came down correctly or not,
even if STATUS tells you that the download succeeded... So. I would
say if you don't know the expected hash, then you can't trust the
source of the download. I personally would not use file(DOWNLOAD to
grab anything of importance unless I knew what it's expected hash is.
May be STATUS is usable in this case as well:
If STATUS var is specified the status of the operation will be put in
var. The status is returned in a list of length 2. The first element
is the numeric return value for the operation, and the second element
is a string value for the error. A 0 numeric error means no error in
the operation
--
Erk
Le gouvernement représentatif n'est pas la démocratie --
http://www.le-message.org
Yes, you can see how file(DOWNLOAD is used from ExternalProject by
inspecting one of its generated download scripts. Here's an example
for downloading the tarball for the zlib project:
message(STATUS downloading...
src='http://zlib.net/zlib-1.2.7.tar.gz'
dst='/Users/davidcole/Downloads/zlib-1.2.7.tar.gz'
timeout='none')
file(DOWNLOAD
http://zlib.net/zlib-1.2.7.tar.gz;
/Users/davidcole/Downloads/zlib-1.2.7.tar.gz
SHOW_PROGRESS
EXPECTED_MD5;60df6a37c56e7c1366cca812414f7b85
# no TIMEOUT
STATUS status
LOG log)
list(GET status 0 status_code)
list(GET status 1 status_string)
if(NOT status_code EQUAL 0)
message(FATAL_ERROR error: downloading
'http://zlib.net/zlib-1.2.7.tar.gz' failed
status_code: ${status_code}
status_string: ${status_string}
log: ${log}
)
endif()
message(STATUS downloading... done)
Plus. if you use the EXPECTED_MD5 argument, then file(DOWNLOAD can
short-circuit/by-pass the network activity and associated wait time if
the file already exists and already has the expected hash value... So
it saves gobs and gobs of time for people executing the same
file(DOWNLOAD command repeatedly.
:-)
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] file( DOWNLOAD ) problem
On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey rcdailey.li...@gmail.com wrote: To do MD5 checks, I need to somehow record the expected MD5 somewhere, which isn't very maintainable. I provide a list of third party libraries that CMake should download from a central third party repository here at work. It is a trusted source, because we know it is, so we don't need to verify the MD5. However, if I could request the MD5 first, and then download, then compare the MD5 the server gave me with what I actually downloaded, that would certainly work just to verify the complete file was downloaded. Other than that, I'll have to rely on the status of the operation... but I don't like that the destination file is created prior to any writes being possible by CMake (it can't write anything if no data was received, so why doesn't it create the file once it has a write buffer?) Recording the MD5 somewhere is the only way to have a reasonable re-assurance that what you've asked for is what you're getting from a network operation. It seems to me that it could be made maintainable if you centralize the knowledge of the checksums in a file that is changed whenever any of the downloadable files is changed. I guess we figure it's no use downloading bits over the network if you can't even open a (presumably local) output file for writing... so we try to open the output file for writing first, and if it succeeds, then we start grabbing bits from the network and writing them into the file as we receive them. There is room for improvement in the file(DOWNLOAD implementation, but it is the way it is right now (and will be for 2.8.10 as well...) Start proposing improvements for it now, and submitting patches to make stuff better for 2.8.11 and/or beyond. :-) HTH, David -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
