Hi again! I tried your commit that skips downloading if file is present and it works. However, imho this is still a suboptimal solution.
This is what I came up with. A script that downloads every url from stack setup yaml and all packages from hackage index. Here is the repository: https://github.com/AleXoundOS/haskell-dev-mirror-script . In other words, you get everything you need to develop software in Haskell except remote docs. After everything is downloaded it uses 20GiB of space and takes about 15 minutes to verify files integrity. Not that much. Tbh, I was afraid it will take more space and time. Downloaded files are placed into proper subdirectories in "mirror" directory that is ready to be served by http server. Clients need to get generated config.yaml with all the paths adjusted to the structure of "mirror" directory. And finally, to bypass the well-known stack bug ignoring "setup-info" field, clients need the generated stack setup yaml for "stack setup" operations. On Sunday, February 19, 2017 at 6:27:56 PM UTC+3, Neil Mitchell wrote: > > > The name of github repository "offline-stack" is a bit misleading. And > > repository descriptions "Install Stack without internet access", "demo > of > > using Haskell Stack without internet access" confuse further. > > Sorry, I'll tweak the text. It's meant to be a demo of the steps you > need to install stack without internet access - not a ready rolled > solution. In particular, it isn't the latter because: > > 1) If your machine has full internet access then this solution isn't > required. You have to have some way to initially get the files onto > the machine, and that will vary by location, so there's no chance of > doing something standard. > > 2) Most places have existing file serving servers, and in my case I > just reused one of them. No point firing up a new Haskell server just > to serve up a few files, when putting it on an Apache/IIS server is > probably easier. > > > I thought that the http-redirect server is able to run offline, i.e. > without > > internet access at all once all the required files were downloaded by > > passing requests through proxy. > > You could easily do that - in fact I've just pushed a 1 line change to > do that - > https://github.com/ndmitchell/offline-stack/commit/86c6ff60dc7df9e20363c1e4f4cbcd79a6543d2f > > (entirely untested). > > > Thank you for all your work in this direction. > > So, do you think it is possible to turn this idea into an offline > server? Do > > you see any pitfalls building up such system for production? Like, does > the > > theory of idea need major adjustments in your opinion? > > In my case, step 1 doesn't work, and step 2 is more hassle than > reusing existing solutions. However, if you can get around step 1 > (which it seems like you can), and step 2 is no big deal, then it > should work. > > Thanks, Neil > > > > > On Sunday, February 19, 2017 at 5:46:49 PM UTC+3, Neil Mitchell wrote: > >> > >> The fact that you can't do "stack new" is known, and raised at > >> https://github.com/commercialhaskell/stack/issues/2804. > >> > >> Note that the http-redirect tool I wrote _always_ goes to the > >> internet, it never serves up mirrored content. If you want to run on a > >> server which doesn't have internet access you'll need to modify that > >> code, or spin up another type of server to serve up the content. The > >> purpose of http-redirect is to robustly test my configuration steps - > >> I didn't write it with the idea of production use in mind. > >> > >> Thanks, Neil > >> > >> > >> On Sun, Feb 19, 2017 at 11:57 AM, Alexander T <alexo...@gmail.com> > wrote: > >> > Ah, also forgot to clarify that the nightly-2017-02-02.yaml file was > >> > already > >> > previously retrieved by the http-redirect server and actually exists > in > >> > it's > >> > mirror directory (with filename renamed according to server's > >> > conventions). > >> > > >> > Server side > >> > $ stat > >> > > >> > > mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml > > >> > File: > >> > > >> > > mirror/https_raw.githubusercontent.com_fpco_lts-haskell_master__lts-8.0.yaml > > >> > Size: 5377585 Blocks: 10504 IO Block: 4096 regular > file > >> > Device: 2dh/45d Inode: 5819 Links: 1 > >> > Access: (0644/-rw-r--r--) Uid: ( 1004/ alex) Gid: ( 100/ > users) > >> > Access: 2017-02-17 05:02:18.946927356 +0300 > >> > Modify: 2017-02-17 05:02:18.936927354 +0300 > >> > Change: 2017-02-19 03:30:53.923441888 +0300 > >> > Birth: - > >> > > >> > On Sunday, February 19, 2017 at 4:29:49 AM UTC+3, Alexander T wrote: > >> >> > >> >> Yes I did. > >> >> > >> >> Server side > >> >> $ env | grep STACK_ROOT > >> >> STACK_ROOT=/data/srv/stack_root > >> >> $ pwd > >> >> /data/srv > >> >> $ ls > >> >> http-redirect mirror stack_root > >> >> $ stack path | grep config > >> >> config-location: /data/srv/stack_root/global-project/stack.yaml > >> >> $ cat /data/srv/stack_root/config.yaml > >> >> urls: > >> >> latest-snapshot: > >> >> http://localhost:3000/https/www.stackage.org/download/snapshots.json > >> >> lts-build-plans: > >> >> > >> >> > http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/ > > >> >> nightly-build-plans: > >> >> > >> >> > http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/ > > >> >> > >> >> package-indices: > >> >> - name: Hackage > >> >> download-prefix: > >> >> > >> >> > http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/ > >> >> http: > >> >> > >> >> > http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz > > >> >> > >> >> > >> >> Client side > >> >> $ env | grep STACK_ROOT > >> >> $ cat ~/.stack/config.yaml > >> >> urls: > >> >> latest-snapshot: > >> >> http://localhost:3000/https/www.stackage.org/download/snapshots.json > >> >> lts-build-plans: > >> >> > >> >> > http://localhost:3000/https/raw.githubusercontent.com/fpco/lts-haskell/master/ > > >> >> nightly-build-plans: > >> >> > >> >> > http://localhost:3000/https/raw.githubusercontent.com/fpco/stackage-nightly/master/ > > >> >> > >> >> package-indices: > >> >> - name: Hackage > >> >> download-prefix: > >> >> > >> >> > http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/package/ > >> >> http: > >> >> > >> >> > http://localhost:3000/https/s3.amazonaws.com/hackage.fpcomplete.com/00-index.tar.gz > > >> >> > >> >> When I try to run "stack path" on client side without internet > >> >> connection > >> >> on both sides > >> >> $ stack path > >> >> Downloading nightly-2017-02-02 build plan ...RedownloadFailed > Request { > >> >> host = "localhost" > >> >> port = 3000 > >> >> secure = False > >> >> requestHeaders = [] > >> >> path = > >> >> > >> >> "/https/ > raw.githubusercontent.com/fpco/stackage-nightly/master//nightly-2017-02-02.yaml" > > > >> >> queryString = "" > >> >> method = "GET" > >> >> proxy = Nothing > >> >> rawBody = False > >> >> redirectCount = 10 > >> >> responseTimeout = ResponseTimeoutDefault > >> >> requestVersion = HTTP/1.1 > >> >> } > >> >> "/home/alex/.stack/build-plan/nightly-2017-02-02.yaml" (Response > >> >> {responseStatus = Status {statusCode = 500, statusMessage = > "Internal > >> >> Server > >> >> Error"}, responseVersion = HTTP/1.0, responseHeaders = > [("Date","Sun, > >> >> 19 Feb > >> >> 2017 01:23:48 > >> >> GMT"),("Server","Warp/3.2.11"),("Content-Type","text/plain; > >> >> charset=utf-8")], responseBody = (), responseCookieJar = CJ {expose > = > >> >> []}, > >> >> responseClose' = ResponseClose}) > >> >> > >> >> At the same time http-redirect server throws: > >> >> HttpExceptionRequest Request { > >> >> host = "raw.githubusercontent.com" > >> >> port = 443 > >> >> secure = True > >> >> requestHeaders = [] > >> >> path = > >> >> "/fpco/stackage-nightly/master//nightly-2017-02-02.yaml" > >> >> queryString = "" > >> >> method = "GET" > >> >> proxy = Nothing > >> >> rawBody = False > >> >> redirectCount = 10 > >> >> responseTimeout = ResponseTimeoutDefault > >> >> requestVersion = HTTP/1.1 > >> >> } > >> >> (ConnectionFailure getAddrInfo: does not exist (Name or service not > >> >> known)) > >> >> > >> >> On Friday, February 17, 2017 at 9:48:46 AM UTC+3, Neil Mitchell > wrote: > >> >>> > >> >>> > Thank you. I tried this, http-redirect server builds fine and > runs. > >> >>> > This is > >> >>> > what I figured out. > >> >>> > >> >>> > Once I did "stack setup", "stack build" through the proxy (with > >> >>> > corresponding config.yaml), I can reproduce it on the same system > >> >>> > without > >> >>> > internet connection. > >> >>> > But if I take a fresh system with empty global stack root (e.g. > >> >>> > ~/.stack on > >> >>> > GNU/Linux) except solely config.yaml, it immediately fails. > >> >>> > http-redirect > >> >>> > throws this: > >> >>> > >> >>> Did you copy the config.yaml from the offline-stack repo into > >> >>> STACK_ROOT? Is your STACK_ROOT actually where you think it is? (use > >> >>> "stack path" to see). > >> >>> > >> >>> > "stack new" fails in both cases with: > >> >>> > Here I can see that this request is not processed by the > >> >>> > http-redirect, > >> >>> > it > >> >>> > keeps silent when stack throws this exception. > >> >>> > > >> >>> > Maybe I missed something? > >> >>> > >> >>> stack new isn't supported - I imagine you have to have another > >> >>> redirect in config.yaml. Pull requests welcome! I've also raised a > >> >>> ticket at https://github.com/ndmitchell/offline-stack/issues/1. > >> >>> > >> >>> Thanks, Neil > >> >>> > >> >>> > > >> >>> > On Wednesday, February 15, 2017 at 11:04:36 PM UTC+3, Neil > Mitchell > >> >>> > wrote: > >> >>> >> > >> >>> >> > Please, can you describe all operations in detail. > >> >>> >> > >> >>> >> Describing is fragile. I've created a github project that > describes > >> >>> >> it. See the README at > >> >>> >> https://github.com/ndmitchell/offline-stack#readme, plus the > code > >> >>> >> which executes it at > >> >>> >> > >> >>> >> > https://github.com/ndmitchell/offline-stack/blob/master/appveyor.yml. > >> >>> >> I test those steps on Appveyor, so it's guaranteed to work/ > >> >>> >> > >> >>> >> > The whole idea of using haskell with stack/cabal offline is > >> >>> >> > crucial > >> >>> >> > for > >> >>> >> > work > >> >>> >> > at many companies. > >> >>> >> > First of all it's not clear how to obtain all the data needed > for > >> >>> >> > stack > >> >>> >> > and > >> >>> >> > cabal to serve locally. > >> >>> >> > >> >>> >> Agreed! Hopefully everything is now 100% clear. > >> >>> >> > >> >>> >> > > >> >>> >> > On Friday, February 10, 2017 at 10:39:51 AM UTC+3, Neil > Mitchell > >> >>> >> > wrote: > >> >>> >> >> > >> >>> >> >> Hi David, > >> >>> >> >> > >> >>> >> >> Its not that bad, now the steps are known, so go ahead and > email > >> >>> >> >> me > >> >>> >> >> if > >> >>> >> >> you > >> >>> >> >> get stuck. Steps are: > >> >>> >> >> > >> >>> >> >> * create stack root > >> >>> >> >> > >> >>> >> >> * copy config.yaml to the root > >> >>> >> >> > >> >>> >> >> * run stack setup with the setup yaml flag > >> >>> >> >> > >> >>> >> >> Figuring our these steps took weeks :) > >> >>> >> >> > >> >>> >> >> Thanks, Neil > >> >>> >> >> > >> >>> >> >> > >> >>> >> >> > >> >>> >> >> On Thu, 9 Feb 2017 at 11:47 pm, David Sicilia > >> >>> >> >> <dpsi...@gmail.com> > >> >>> >> >> wrote: > >> >>> >> >>> > >> >>> >> >>> Hi Neil, > >> >>> >> >>> > >> >>> >> >>> Thank you for raising the ticket, I just gave it a +1, if > that > >> >>> >> >>> helps > >> >>> >> >>> :-) > >> >>> >> >>> > >> >>> >> >>> So would you recommend that I go ahead and try it right now, > >> >>> >> >>> or is it complicated enough that I should just wait for that > >> >>> >> >>> change > >> >>> >> >>> to be implemented? I guess what I mean is, after you > realized > >> >>> >> >>> which config files to change, was it straightforward from > >> >>> >> >>> there? > >> >>> >> >>> > >> >>> >> >>> David > >> >>> >> >>> > >> >>> >> >>> On Saturday, January 21, 2017 at 11:27:22 AM UTC-5, David > >> >>> >> >>> Sicilia > >> >>> >> >>> wrote: > >> >>> >> >>>> > >> >>> >> >>>> Hi there, > >> >>> >> >>>> > >> >>> >> >>>> > >> >>> >> >>>> I am behind a firewall with strict rules about downloading > >> >>> >> >>>> software > >> >>> >> >>>> from > >> >>> >> >>>> the internet, > >> >>> >> >>>> > >> >>> >> >>>> so I would like to know if it is possible to download an > >> >>> >> >>>> entire > >> >>> >> >>>> stackage > >> >>> >> >>>> LTS snapshot > >> >>> >> >>>> > >> >>> >> >>>> in one go and then serve it on a local server, to which we > >> >>> >> >>>> could > >> >>> >> >>>> then > >> >>> >> >>>> point the stack tool. > >> >>> >> >>>> > >> >>> >> >>>> > >> >>> >> >>>> By "snapshot" I would be referring to the source code for > all > >> >>> >> >>>> packages > >> >>> >> >>>> because > >> >>> >> >>>> > >> >>> >> >>>> we'd need to draw from that local server to build on both > >> >>> >> >>>> Linux > >> >>> >> >>>> and > >> >>> >> >>>> Windows. > >> >>> >> >>>> > >> >>> >> >>>> > >> >>> >> >>>> For example, perhaps we'd be able to go to a server within > the > >> >>> >> >>>> firewall > >> >>> >> >>>> and run > >> >>> >> >>>> > >> >>> >> >>>> "stack new", then edit the yaml file to point it to a local > >> >>> >> >>>> stackage > >> >>> >> >>>> server with given > >> >>> >> >>>> > >> >>> >> >>>> resolver number, then it would just behave normally from > then > >> >>> >> >>>> on, > >> >>> >> >>>> except > >> >>> >> >>>> always > >> >>> >> >>>> > >> >>> >> >>>> downloading packages from the local server. > >> >>> >> >>>> > >> >>> >> >>>> > >> >>> >> >>>> What about compilers? I know that stack also downloads the > >> >>> >> >>>> compilers, > >> >>> >> >>>> so > >> >>> >> >>>> I guess > >> >>> >> >>>> > >> >>> >> >>>> those would have to be included in the downloaded > >> >>> >> >>>> snapshot-package > >> >>> >> >>>> for > >> >>> >> >>>> various > >> >>> >> >>>> > >> >>> >> >>>> platforms? > >> >>> >> >>>> > >> >>> >> >>>> > >> >>> >> >>>> Any guidance would be appreciated, because we will not be > able > >> >>> >> >>>> to > >> >>> >> >>>> make > >> >>> >> >>>> use of > >> >>> >> >>>> > >> >>> >> >>>> stack/stackage without something like this! > >> >>> >> >>>> > >> >>> >> >>>> > >> >>> >> >>>> Thank you > >> >>> >> >>>> > >> >>> >> >>>> David > >> >>> >> >>> > >> >>> >> >>> -- > >> >>> >> >>> You received this message because you are subscribed to the > >> >>> >> >>> Google > >> >>> >> >>> Groups > >> >>> >> >>> "haskell-stack" group. > >> >>> >> >>> To unsubscribe from this group and stop receiving emails > from > >> >>> >> >>> it, > >> >>> >> >>> send > >> >>> >> >>> an > >> >>> >> >>> email to haskell-stac...@googlegroups.com. > >> >>> >> >>> To post to this group, send email to > >> >>> >> >>> haskel...@googlegroups.com. > >> >>> >> >>> To view this discussion on the web visit > >> >>> >> >>> > >> >>> >> >>> > >> >>> >> >>> > >> >>> >> >>> > https://groups.google.com/d/msgid/haskell-stack/35616520-204d-41f6-b1c9-e37ee1570473%40googlegroups.com. > > > >> >>> >> >>> For more options, visit https://groups.google.com/d/optout. > >> >>> >> > > >> >>> >> > -- > >> >>> >> > You received this message because you are subscribed to the > >> >>> >> > Google > >> >>> >> > Groups > >> >>> >> > "haskell-stack" group. > >> >>> >> > To unsubscribe from this group and stop receiving emails from > it, > >> >>> >> > send > >> >>> >> > an > >> >>> >> > email to haskell-stac...@googlegroups.com. > >> >>> >> > To post to this group, send email to > haskel...@googlegroups.com. > >> >>> >> > To view this discussion on the web visit > >> >>> >> > > >> >>> >> > > >> >>> >> > > >> >>> >> > > https://groups.google.com/d/msgid/haskell-stack/094da0e4-6eab-4c10-8842-4d3c37b3ad78%40googlegroups.com. > > > >> >>> >> > > >> >>> >> > For more options, visit https://groups.google.com/d/optout. > >> >>> > > >> >>> > -- > >> >>> > You received this message because you are subscribed to the > Google > >> >>> > Groups > >> >>> > "haskell-stack" group. > >> >>> > To unsubscribe from this group and stop receiving emails from it, > >> >>> > send > >> >>> > an > >> >>> > email to haskell-stac...@googlegroups.com. > >> >>> > To post to this group, send email to haskel...@googlegroups.com. > >> >>> > To view this discussion on the web visit > >> >>> > > >> >>> > > >> >>> > > https://groups.google.com/d/msgid/haskell-stack/674862b8-2d46-4c6e-ac93-0b1cdfde0ec5%40googlegroups.com. > > > >> >>> > > >> >>> > For more options, visit https://groups.google.com/d/optout. > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "haskell-stack" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send > >> > an > >> > email to haskell-stac...@googlegroups.com. > >> > To post to this group, send email to haskel...@googlegroups.com. > >> > To view this discussion on the web visit > >> > > >> > > https://groups.google.com/d/msgid/haskell-stack/17cc8023-090c-4cf8-982e-c8e743df65ed%40googlegroups.com. > > > >> > > >> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "haskell-stack" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to haskell-stac...@googlegroups.com <javascript:>. > > To post to this group, send email to haskel...@googlegroups.com > <javascript:>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/haskell-stack/c2b35b88-e1f7-4a87-adf1-0ac423a4d53a%40googlegroups.com. > > > > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "haskell-stack" group. To unsubscribe from this group and stop receiving emails from it, send an email to haskell-stack+unsubscr...@googlegroups.com. To post to this group, send email to haskell-stack@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/haskell-stack/47458241-6f61-4cc8-8357-f0275d179c9b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
