Dear Cabal maintainers,
A couple of days ago I was unable to cabal-install a library from
Hackage -- 00-index.tar.gz was not downloaded completely. And if
obtaining one with wget, other .tar.gz files (packages) still could not
be fetched in full.
This problem has been reported already (see
http://hackage.haskell.org/trac/hackage/ticket/562).
06/11/09 09:49:25 changed by duncan:
> Why has this started cropping up all of a sudden? Never seen this
> before then 3 reports in as many days. Do we suspect HTTP-4000.0.6 ->
> 7 perhaps?
06/11/09 13:02:47 changed by michaeldever:
> So it's definitely a problem with the HTTP package in my opinion. I'm
> not sure if it is a problem with the packages proxy handling, as it
> does download some of the package, but not all of it.
>
> Seeing as both the Zlib library, and tar yield an end of stream error,
> its something that I'm reckoning is happening during transport.
06/13/09 09:05:26 changed by michaeldever:
> http://trac.haskell.org/http/ticket/8#comment:1
The bug is not in HTTP API, but in the way cabal-install uses it.
The type of HTTP response body is polymorphic within `HTTP' library
(rspBody :: Response a -> a) but it is specialized to Lazy.ByteString by
cabal-install's `getHTTP' function (Distribution/Client/HttpUtils.hs).
Once the type of response body is changed to _strict_ ByteString, files
get downloaded through proxy completely.
The attached module [proxy-POC.hs] makes this quite apparent (you need
to be behind a proxy; HTTP >= 4000.0.8):
v...@takeshi:~/src$ time runhaskell proxy-POC.hs
Content-Length: 1200593
bytes downloaded: 2408
proxy-POC.hs: user error (sizes differ)
real 0m1.210s
user 0m0.556s
sys 0m0.052s
v...@takeshi:~/src$ time runhaskell -DSTRICT proxy-POC.hs
Content-Length: 1200593
bytes downloaded: 1200593
real 0m17.956s
user 0m0.620s
sys 0m0.028s
v...@takeshi:~/src$ runhaskell proxy-POC.hs # repeatable
Content-Length: 1200593
bytes downloaded: 2408
proxy-POC.hs: user error (sizes differ)
There are only 4 lines that need to be changed (2 in HttpUtils.hs and 2
in Fetch.hs); see the accompanying patch.
...And could anyone explain me, why don't lazy ByteString cause cropped
downloads in proxy-free environment?
Thank you.
--
vvv
diff -ru /tmp/cabal-install-0.6.2/Distribution/Client/Fetch.hs ./Distribution/Client/Fetch.hs
--- /tmp/cabal-install-0.6.2/Distribution/Client/Fetch.hs 2009-02-19 15:07:52.000000000 +0200
+++ ./Distribution/Client/Fetch.hs 2009-09-18 18:34:41.989206160 +0300
@@ -73,6 +73,7 @@
( Response(..) )
import Network.Stream
( ConnError(..) )
+import Data.ByteString.Lazy ( fromChunks )
downloadURI :: Verbosity
@@ -89,7 +90,7 @@
Right rsp
| rspCode rsp == (2,0,0)
-> do info verbosity ("Downloaded to " ++ path)
- writeFileAtomic path (rspBody rsp)
+ writeFileAtomic path $ fromChunks [rspBody rsp]
--FIXME: check the content-length header matches the body length.
--TODO: stream the download into the file rather than buffering the whole
-- thing in memory.
diff -ru /tmp/cabal-install-0.6.2/Distribution/Client/HttpUtils.hs ./Distribution/Client/HttpUtils.hs
--- /tmp/cabal-install-0.6.2/Distribution/Client/HttpUtils.hs 2009-02-19 15:07:52.000000000 +0200
+++ ./Distribution/Client/HttpUtils.hs 2009-09-18 18:35:34.722456689 +0300
@@ -15,8 +15,8 @@
, setOutHandler, setErrHandler, setProxy, request)
import Control.Monad
( mplus, join, liftM2 )
-import qualified Data.ByteString.Lazy as ByteString
-import Data.ByteString.Lazy (ByteString)
+import qualified Data.ByteString as ByteString
+import Data.ByteString (ByteString)
#ifdef WIN32
import System.Win32.Types
( DWORD, HKEY )
{-# LANGUAGE CPP #-}
{-# OPTIONS_GHC -Wall #-}
-- | Re: cabal-install update fails going through a HTTP proxy
-- [hackage ticket #562]
--
-- See http://hackage.haskell.org/trac/hackage/ticket/562#comment:9
--
module Main where
import Network.HTTP.Proxy (fetchProxy)
import Network.HTTP (Request(..), Response(..), Header(..), RequestMethod(..),
HeaderName(..))
import Network.HTTP.Headers (lookupHeader)
import Network.Browser (browse, request, setProxy, setOutHandler, setErrHandler)
import Network.Stream (Result)
import System.Environment (getArgs)
import Network.URI (URI(..), parseURI)
import Distribution.Simple.Utils (warn, debug)
import Distribution.Verbosity (normal, {-deafening,-} Verbosity)
import Data.Maybe (fromMaybe)
import Control.Monad (when)
--- Problems (with downloading via a proxy) disappear once strict
--- ByteStrings are used. Thus the fix is to remove `.Lazy' suffixes.
-- #define STRICT
#ifdef STRICT
import qualified Data.ByteString as BS
import Data.ByteString (ByteString)
#else
import qualified Data.ByteString.Lazy as BS
import Data.ByteString.Lazy (ByteString)
#endif
{-----------------------------------------------------------------------
v...@takeshi:~/src$ time runhaskell proxy-POC.hs
Content-Length: 1200593
bytes downloaded: 2408
proxy-POC.hs: user error (sizes differ)
real 0m1.210s
user 0m0.556s
sys 0m0.052s
v...@takeshi:~/src$ time runhaskell -DSTRICT proxy-POC.hs
Content-Length: 1200593
bytes downloaded: 1200593
real 0m17.956s
user 0m0.620s
sys 0m0.028s
-----------------------------------------------------------------------}
main :: IO ()
main = do
args <- getArgs
rsp <- fetch $ (args ++ ["http://hackage.haskell.org/packages/archive/";
++ "00-index.tar.gz"]) !! 0
let lenHeader = fromMaybe "" $ lookupHeader HdrContentLength (rspHeaders rsp)
lenBody = show $ BS.length $ rspBody rsp
report lenHeader lenBody
report :: String -> String -> IO ()
report hdr bdy = do
when (avail hdr) (putStrLn $ "Content-Length: " ++ hdr)
putStrLn ("bytes downloaded: " ++ bdy)
when (avail hdr && hdr /= bdy) (fail "sizes differ")
where avail = not . null
fetch :: String -> IO (Response ByteString)
fetch s = do
case parseURI s of
Nothing -> fail ("fetch: unable to parse URI: " ++ s)
Just u -> do
Right rsp <- getHTTP normal u
return rsp
------------------------------------------------------------------------
-- The following functions are copied (with minimal changes) from
-- cabal-install's 'Distribution.Client.HttpUtils'.
-- |Carry out a GET request, using the local proxy settings
getHTTP :: Verbosity -> URI -> IO (Result (Response ByteString))
getHTTP verbosity uri = do
-- p <- proxy verbosity
p <- fetchProxy False
let req = mkRequest uri
(_, resp) <-
browse $ do
setErrHandler (warn verbosity . ("http error: "++))
setOutHandler (debug verbosity)
setProxy p
request req
return (Right resp)
mkRequest :: URI -> Request ByteString
mkRequest uri = Request{ rqURI = uri
, rqMethod = GET
, rqHeaders = [Header HdrUserAgent userAgent]
++ [Header HdrCacheControl "no-cache"] -- XXX *new*
, rqBody = BS.empty }
-- where userAgent = "cabal-install/" ++ display Paths_cabal_install.version
where userAgent = "proxy-POC.hs"
_______________________________________________
cabal-devel mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/cabal-devel
