Apologies for potential confusion from the link I sent. While we don't
think our problem was a DDoS, there were other aspects of the thread
that caused concern on my part -- largely discussion of support levels
and stories about poor hardware. Given that they've now done some work
on hardware I'm quite willing to stick with Hetzner for the time being
and hope this doesn't repeat (again).
Work on backups, fallovers, and hackage mirrors should continue still,
I think, and Austin's proposal seems the right way to go.
Paul -- we'll let you know if we actually have any concrete ideas/plans
regarding Abbot. I'm just jumping the gun/ speculating/exploring ideas
at the moment. OSU says for colos:
"We require all servers to have sliding rails and cable management arms.
Hardware must be purchased from a vendor rather than built by hand to
ensure it operates as intended."
And also that they prefer colos to be no bigger than 2U. Having never
physically seen abbott I don't know if it qualifies anyway?
From the standpoint of you and galois, would you prefer that you no
longer had to worry about abbot, or prefer that you didn't have to worry
about moving abbot? That would be a consideration too :-)
--Gershom
On 11/17/13 8:02 AM, Austin Seipp wrote:
The problem was not DDoS, I think. Our speculation is that it might
have been a software/hardware issue possibly (kernel panic, a total
OOM and grinding to a halt is also possible I suppose.)
Hetzner replaced the RAM and upgraded the BIOS on the machine as it
was not responding to any keystrokes in DC 19. It is now running
smoothly again, but who knows when it might strike.
In light of this, using Rackspace resources (they graciously donated
free cloud services, up to $2000 USD a month at the moment,) I have
rebuilt a new backup machine running FreeBSD 9.2 using ZFS. It has 4TB
of space, 16GB of RAM and generally should be enough. This is a
serious just-in-case, and I will be rewriting the backup scripts to
use this machine quite soon, and give the other admins access (sadly I
think backups may have silently broken at one point, but I'm not 100%
sure.)
I'll also be looking to use Rackspace to host an official hackage
mirror. To do this safely we'd technically need package signing in
place, but it could also be a blessed instance (under the haskell.org
domain w/ SSL enabled) for the moment, outside of Hetzner and in a
different availability zone (say, the US or Australia.) This should
also mitigate the impact for a large variety of users in the wild.
I've CC'd Duncan so he's aware (although I'm sure he's on this list.)
We'll also still have enough capacity to run GHC buildbots too.
Finally, I'm also willing to continue with Hetzner and they have
worked wonderfully for me personally, and also us in the past (lambda
has an uptime of nearly a year!) However, I think splitting up some of
the services is sensible where possible for critical stuff, so I'm
taking the first step to that (off-site backups and a mirror.) Others
can come as needed.
On Sun, Nov 17, 2013 at 6:46 AM, Yitzchak Gale <[email protected]> wrote:
I agree with Mark. Suite Solutions is are using Hetzner extensively
and we are very happy with them. It's a tremendous value.
Thanks to Gershom for the link to the Y thread though. I'll make
sure that our IT dept. knows to have a contingency plan in
case we are specifically targeted. We should do the same here
in the Haskell community as Jason suggests, unless we feel we
can afford a service that provides what Hetzner provides plus
DDoS protection.
On Sun, Nov 17, 2013 at 6:35 AM, Mark Lentczner
<[email protected]> wrote:
So, I have a hetzner machine, as do others I know in the community - and the
service has been stellar.
The ycombinator thread makes it clear: The problem is DDoS, and of course
Hetzner provides no protection against that - which is expected. Is there
any reason to believe that we will be protected from DDoS at osu?
Do we have reason to believe that this is a Hetzner problem? Do we know the
root cause of our going down?
- Mark
_______________________________________________
haskell-infrastructure mailing list
[email protected]
http://community.galois.com/mailman/listinfo/haskell-infrastructure
_______________________________________________
haskell-infrastructure mailing list
[email protected]
http://community.galois.com/mailman/listinfo/haskell-infrastructure
_______________________________________________
haskell-infrastructure mailing list
[email protected]
http://community.galois.com/mailman/listinfo/haskell-infrastructure
