Re: [CODE4LIB] code4lib.org hosting
On 8/1/07, D Chudnov [EMAIL PROTECTED] wrote: Separately I can bring up an emergency/temporary backup of the www.code4lib.org site if it is not otherwise possible before the end of the week. I'm in transit all day tomorrow, but let me know by first thing friday Sounds like a great idea to me. You can get the backup files from Ross or Ed, I imagine. I'm not sure why this hasn't been suggested sooner since the framework is already on your machine from the last failure. As to where the production code4lib should go, OSU or iBiblio, I don't really have an opinion. Seems like six of one and a half dozen of the other to me. Those who have strong opinions should probably post them here for discussion (I think the in channel 'votes' were viewed by all as preliminary). As Ross said, I think everyone was happy with OSU just because working code wins -- i.e., it was here and now. Kevin
Re: [CODE4LIB] code4lib.org hosting
On 8/2/07, Kevin S. Clarke [EMAIL PROTECTED] wrote: On 8/1/07, D Chudnov [EMAIL PROTECTED] wrote: Separately I can bring up an emergency/temporary backup of the www.code4lib.org site if it is not otherwise possible before the end of the week. I'm in transit all day tomorrow, but let me know by first thing friday Sounds like a great idea to me. You can get the backup files from Ross or Ed, I imagine. I'm not sure why this hasn't been suggested sooner since the framework is already on your machine from the last failure. I would have suggested it sooner but most of my attention has been in moving boxes. And I didn't realize nobody else was pushing on it other than we'll see how anvil looks when can get to it. I'm don't have a beef with that per se - I just think we need the main site back up asap, even if only temporarily hosted. -Dan
Re: [CODE4LIB] code4lib.org hosting
Not make this full-contact Ross/Dan ping-pong, but here we go. On 8/1/07, D Chudnov [EMAIL PROTECTED] wrote: 1) we could have found someplace 2) this is a problem either way 3) this is a problem either way 4) a few days' downtime is one thing - a few weeks' is another. I suppose these are really easy to say but I still haven't seen you asking how you can help or what you can do get this back up since the incident (until now). In fact, nobody has. Until Jeremy's (perfectly reasonable) point that the C4L08 planning is there, nobody has even indicated that this is more than an inconvenience (short of the journal site - but even then, we've come up with alternatives). I mean, it's one thing to say that 'we need to have policies and procedures in case of emergency', it's another thing to actually create them and approve them and implement them. Then there's the fact that it's quite likely that our drupal instance is what was the cause of the break-in in the first place, meaning that would have to be addressed or the mirror site is just as vulnerable. I guess my point is, I don't see much point in criticizing the all-volunteer effort and donated server space that has gone into code4lib.org now. Let's set up a second backup of your backup? I'll take a copy, and add another to s3 for safer-keeping. Ask Ed for them. Nobody is stopping me. I was offline all weekend, and busy otherwise, and it wasn't clear to me whether this was a done deal already. Oregon State is making a proposal. Other proposals are welcome. I would have suggested it sooner but most of my attention has been in moving boxes. And Kevin has been on vacation. And I have been working on my bathroom. And... You see? Nobody is in charge of this or responsible. Being complainy is counterproductive, honestly. If we need to fix the governance issues, then let's do it. But don't assume they'll be taken care of yesterday. -Ross. -Ross.
Re: [CODE4LIB] code4lib.org hosting
On 8/1/07, D Chudnov [EMAIL PROTECTED] wrote: 1. Why isn't the www.code4lib.org site already back up (at minimum)? When the server became unavailable during c4lc 2007, we were able to restore from offsite backups and have an emergency-mode snapshot live and useful within a matter of a few hours at most. It seems this could have been done within a few days if there had been offsite backups available. Because I haven't had time...and even if I did I don't want the responsibility anymore. 2. Are there offsite backups of the www.code4lib.org site - its files and database? Yes, although they are partially corrupted because we didn't have a rolling backup 3. The discussion seemed to only involve one proposal. There wasn't a call for any other proposals, and it wasn't clear to me that by missing this meeting (i was at a gathering with several other people with an interest in anvil and other things c4l) that I would miss out on any opportunity to have input. I'd like to propose a different hosting plan. Shouldn't there be a chance for more discussion here? Ryan Eby and Brad LaJeunesse were also approached as well. The announcement went out to the public and there was a reminder...what more can one do? I understand that some of these questions might seem to be coming a bit late, and I'm sorry to be in a position where my jerkiness is all the worse because of it. But I still think these are questions that need answers. Hopefully you've got them now. //Ed
Re: [CODE4LIB] code4lib.org hosting
On 8/2/07, Ed Summers [EMAIL PROTECTED] wrote: 2. Are there offsite backups of the www.code4lib.org site - its files and database? Yes, although they are partially corrupted because we didn't have a rolling backup Ah. Ouch. Okay. I understand that some of these questions might seem to be coming a bit late, and I'm sorry to be in a position where my jerkiness is all the worse because of it. But I still think these are questions that need answers. Hopefully you've got them now. Yep. Thanks.
Re: [CODE4LIB] code4lib.org hosting
D Chudnov wrote: 1. Why isn't the www.code4lib.org site already back up (at minimum)? Because nobody is getting paid to sysadmin that machine? To my mind, it is clear that that machine was not being sysadmined up to professional standards---but absolutely no-one, ever, committed to doing that! (Certainly a volunteer could commit to that too, it does not require being paid--but nobody did). It is a hobby machine. There was no reason to expect that machine to be sysadmined to professional standards. If you expected that, your expectations were mis-placed. Now, this incident made everyone realize that the community in fact did have a need for a host that was admin'ed to professional standards. The OSU proposal was a way of achieving that. The official 'proposal' has not been sent to the list YET, but it will be. If you would like to make or solicit other proposals, you are free to. This is the proposal that came out of the discussion in IRC yesterday--it was the consensus of those in IRC. But not everyone could make the IRC meeting, or maybe not everyone realized they had an interest in it. That's okay, no final decisions were made, a proposal is still forthcoming, and it will be sent to this list. You can still make other proposals if you like. This is just the one that represented the consensus of one IRC discussion. Your concerns may not be misplaced Dan, and your interest is certainly well-placed, but your anger is misplaced. Jonathan When the server became unavailable during c4lc 2007, we were able to restore from offsite backups and have an emergency-mode snapshot live and useful within a matter of a few hours at most. It seems this could have been done within a few days if there had been offsite backups available. 2. Are there offsite backups of the www.code4lib.org site - its files and database? 3. The discussion seemed to only involve one proposal. There wasn't a call for any other proposals, and it wasn't clear to me that by missing this meeting (i was at a gathering with several other people with an interest in anvil and other things c4l) that I would miss out on any opportunity to have input. I'd like to propose a different hosting plan. Shouldn't there be a chance for more discussion here? 4. Could somebody please post (to this list) an exact statement of what the current proposal is? 5. Could somebody please post the transcript to the list? I understand that some of these questions might seem to be coming a bit late, and I'm sorry to be in a position where my jerkiness is all the worse because of it. But I still think these are questions that need answers. -Dan -- Jonathan Rochkind Digital Services Software Engineer The Sheridan Libraries Johns Hopkins University 410.516.8886 rochkind (at) jhu.edu
Re: [CODE4LIB] code4lib.org hosting
D Chudnov wrote: 1) we could have found someplace 2) this is a problem either way 3) this is a problem either way 4) a few days' downtime is one thing - a few weeks' is another. Yes, I believe everyone realizes this is a problem. That is why everyone has been talking about figuring out a solution. The OSU proposal was deemed the best available solution yesterday in a discussion, but that doesn't mean you can't propose other ones. If you think you need to convince everyone that the current situation is a problem, I think you are tilting at windmills. Everyone is convinced. Jonathan 2. Are there offsite backups of the www.code4lib.org site - its files and database? Yes, and they live on my machine at work. Let's set up a second backup of your backup? I'll take a copy, and add another to s3 for safer-keeping. 3. The discussion seemed to only involve one proposal. There wasn't a call for any other proposals, and it wasn't clear to me that by missing this meeting (i was at a gathering with several other people with an interest in anvil and other things c4l) that I would miss out on any opportunity to have input. I'd like to propose a different hosting plan. Shouldn't there be a chance for more discussion here? Who's stopping you? We announced a 'town hall meeting', nobody publicly dissented on the mailing list. We're 'discussing' now, you're complaining but not articulating an alternative. You mentioned ibiblio in channel, but until something tangible is offered, the Oregon State offer meets the 'one in hand is worth two in the bush' criteria. Nobody is stopping me. I was offline all weekend, and busy otherwise, and it wasn't clear to me whether this was a done deal already. I'm not complaining, I'm just seeking clarification. In a complainy-sounding way. I understand that some of these questions might seem to be coming a bit late, and I'm sorry to be in a position where my jerkiness is all the worse because of it. But I still think these are questions that need answers. I don't think anything has been 'decided'. We had a meeting, OSU stepped forward, nobody present objected. www.code4lib.org is still 'down'. I propose that we move hosting of www.code4lib.org to ibiblio if they'll have us. They've been there for 15(+?) years, they are there for exactly this purpose, and they're not for-profit. I've had good luck hosting things there, and they're liberal about accounts, so long as you don't prove to be an idiot. To support this I'd be happy to sign up for support duties. Separately I can bring up an emergency/temporary backup of the www.code4lib.org site if it is not otherwise possible before the end of the week. I'm in transit all day tomorrow, but let me know by first thing friday -Dan -- Jonathan Rochkind Digital Services Software Engineer The Sheridan Libraries Johns Hopkins University 410.516.8886 rochkind (at) jhu.edu
Re: [CODE4LIB] code4lib.org hosting
Dan convinces me that iBiblio might be better than OSU, in terms of their type of organization. But OSU has made an offer of free supported hosting, under terms that to me seem quite acceptable. Unless iBiblio makes a similar offer under equal or better terms... I still think that an imperfect solution (we all know about what happens when you insist on perfection, right?) is better than the current situation. Dan has in fact articulated quite well what is wrong with the current situation. So Dan and everyone else agrees that it is desirable for the current situation to change. The OSU proposal is the best thing on the table right now for an improvement. If Dan or anyone else wants to spend time on putting together another possibility, that would certainly be welcome. If not, nothing is permanent, we can always change again later (although of course it's a pain and we should not plan on doing such). Jonathan Ross Singer wrote: Thanks, Dan. This is actually the best argument against an arrangement outside of organization that explicitly does this sort of work. This much more soundly articulates my concern (I was using university counsel as an example, but anyone in the chain can potentially disrupt this entire community for whatever their reason). Ed and I actually shared this concern (well, I did and Ed was probably idle and wasn't disagreeing). We saw something similar recently: John Blyberg had offered a similar sort of hosting service at AADL. I asked him about what would happen to said service if, on the odd chance, he were to leave. He was rather vague about it, but said it would be the responsibility of his successor. About two weeks later he announced his resignation and there has been nothing about this (that I know of) since. Maybe Ryan Eby has more info here. This isn't a criticism of Blyberg, AADL or good intentions. It's just reality. And I think it illustrates the point perfectly. -Ross. On 8/2/07, D Chudnov [EMAIL PROTECTED] wrote: On 8/2/07, Jeremy Frumkin [EMAIL PROTECTED] wrote: So, the one point that I don't understand Dan is what you see as the advantage of having c4l hosted at ibiblio is as opposed to OSU? I don't believe there are any technical advantages (we already support most of the current c4l software, such as Drupal and trac, here at OSU, and whatever accounts / access is requested by the c4l community will be available). Let me ask directly – is there a continued concern about having a university donate hosting services? Yes, I have a concern about a university hosting arrangement like what you've described, though it doesn't have anything to do with OSU or you. :) ibiblio is in the business of being a neutral hosting space for sites like this (and, yes, they're at a university, but, their long-lived project there is long-dedicated to this). Special arrangements to host code4lib.org at any particular university through the good graces of one or two people like you are just that - special arrangements, predicated on a generally favorable situation and the kindness of a couple of supportive individuals both in our community - presumably you and rordway - and at OSU (your boss and network/server support staff). We've all seen many job postings floating by this list, and many of us have changed jobs in recent months/years, and even when people don't change jobs, the circumstances behind special arrangements change for other reasons. A new network guy comes in who doesn't like you, somebody else changes a server without caring how it affects your thing, a boss wants you to focus more on internal tasks, you lose interest yourself - I'm just talking from my own experience here, this is just what happens over time. And over time, what you want is a host that gets that you're hosting something on a voluntary basis that just needs a stable place to live, and is going to honor that your collection of stuff, whatever it is, is somehow important enough that they're going to go out of their way not to clobber it, or suddenly decide that they don't want it anymore. My experience with ibiblio is that they do that. My experience with OSU only indicates that it'd a good place to host stuff too, and I'd bet that everybody here would trust you and Ryan to do it right - again, that's not my concern. And, obviously, the linux kernel people seem to think OSU is a good place to host stuff, among others, so that's not a problem either. :) I just think we should take the opportunity to move to an organization directly focused on exactly what we need from a host. The other issue that worries me is that at a site like ibiblio, direct communications with the hosting provider could be managed through any of a number of site admins - unlike with a particular university, where if there's a serious issue, it would have to go through the one or two code4libbers at that university to get to the server people to actually pay attention. Fwiw, I don't worry at all about
Re: [CODE4LIB] code4lib.org hosting
On 8/2/07, D Chudnov [EMAIL PROTECTED] wrote: I thought the site would be back up in some state earlier this week, and like you I've been busy in the meantime. Seeing that it was still down after 10 days led me to want to say something and offer to help. Well, to be fair, some people have stepped up and taken ownership of 'parts' of the site. Bill Denton. Jason Ronallo. Hands haven't simply been sat upon. Drupal and Trac were the two much more difficult instances to migrate and, as I've said previously, nobody has offered to take them. You have now, so that's progress. Still, there needs to be some 'plan' with that, otherwise our 'temporary' home becomes www.code4lib.org's new 'permanent' home out of apathy and negligence. That's more what I'm getting at. I mean, it's one thing to say that 'we need to have policies and procedures in case of emergency', it's another thing to actually create them and approve them and implement them. I didn't say that. No, but the point is that nobody's in charge of this. Either somebody makes an executive decision (and risks stepping on somebody else's toes) or everybody calls 'not it'. Because we don't know where we're heading with this, nobody's really willing to commit a lot of time or energy. I think that needs to be addressed. Then there's the fact that it's quite likely that our drupal instance is what was the cause of the break-in in the first place, meaning that would have to be addressed or the mirror site is just as vulnerable. Nobody knows this for sure, so it's only speculation, and doesn't help. The fact that it's quite likely? Right, well maybe it's not how they got in, but do you want to run it 'as-is' on a machine you're responsible for? Being complainy is counterproductive, honestly. If we need to fix the governance issues, then let's do it. But don't assume they'll be taken care of yesterday. I'm not complaining about governance issues. Did I use the word governance? I want to see the site back up and can help if need be. It's impossible to know if I can help get the site back up if I don't know why it isn't back up already. But these are 'governance' issues. Nobody 'owns' this problem because nobody is in charge. There is no plan forward so anything done is reactionary and temporary. We need a plan and structure around this so we can move past all this. -Ross.
Re: [CODE4LIB] code4lib.org hosting
Thanks, Dan. This is actually the best argument against an arrangement outside of organization that explicitly does this sort of work. This much more soundly articulates my concern (I was using university counsel as an example, but anyone in the chain can potentially disrupt this entire community for whatever their reason). Ed and I actually shared this concern (well, I did and Ed was probably idle and wasn't disagreeing). We saw something similar recently: John Blyberg had offered a similar sort of hosting service at AADL. I asked him about what would happen to said service if, on the odd chance, he were to leave. He was rather vague about it, but said it would be the responsibility of his successor. About two weeks later he announced his resignation and there has been nothing about this (that I know of) since. Maybe Ryan Eby has more info here. This isn't a criticism of Blyberg, AADL or good intentions. It's just reality. And I think it illustrates the point perfectly. -Ross. On 8/2/07, D Chudnov [EMAIL PROTECTED] wrote: On 8/2/07, Jeremy Frumkin [EMAIL PROTECTED] wrote: So, the one point that I don't understand Dan is what you see as the advantage of having c4l hosted at ibiblio is as opposed to OSU? I don't believe there are any technical advantages (we already support most of the current c4l software, such as Drupal and trac, here at OSU, and whatever accounts / access is requested by the c4l community will be available). Let me ask directly – is there a continued concern about having a university donate hosting services? Yes, I have a concern about a university hosting arrangement like what you've described, though it doesn't have anything to do with OSU or you. :) ibiblio is in the business of being a neutral hosting space for sites like this (and, yes, they're at a university, but, their long-lived project there is long-dedicated to this). Special arrangements to host code4lib.org at any particular university through the good graces of one or two people like you are just that - special arrangements, predicated on a generally favorable situation and the kindness of a couple of supportive individuals both in our community - presumably you and rordway - and at OSU (your boss and network/server support staff). We've all seen many job postings floating by this list, and many of us have changed jobs in recent months/years, and even when people don't change jobs, the circumstances behind special arrangements change for other reasons. A new network guy comes in who doesn't like you, somebody else changes a server without caring how it affects your thing, a boss wants you to focus more on internal tasks, you lose interest yourself - I'm just talking from my own experience here, this is just what happens over time. And over time, what you want is a host that gets that you're hosting something on a voluntary basis that just needs a stable place to live, and is going to honor that your collection of stuff, whatever it is, is somehow important enough that they're going to go out of their way not to clobber it, or suddenly decide that they don't want it anymore. My experience with ibiblio is that they do that. My experience with OSU only indicates that it'd a good place to host stuff too, and I'd bet that everybody here would trust you and Ryan to do it right - again, that's not my concern. And, obviously, the linux kernel people seem to think OSU is a good place to host stuff, among others, so that's not a problem either. :) I just think we should take the opportunity to move to an organization directly focused on exactly what we need from a host. The other issue that worries me is that at a site like ibiblio, direct communications with the hosting provider could be managed through any of a number of site admins - unlike with a particular university, where if there's a serious issue, it would have to go through the one or two code4libbers at that university to get to the server people to actually pay attention. Fwiw, I don't worry at all about the inappropriate content issue. Everything the www.code4lib.org site is about is utterly appropriate. But then again, I don't think that any personal sites, or even trac, should be co-located with it, but that's a separate issue I won't argue at length about. (Actually, I'd rather see something like: www.code4lib.org goes to ibiblio, trac.code4lib.org goes to OSU, personal blogs stay on anvil for those who pay into anvil, or wherever else you want to put them.) -Dan
Re: [CODE4LIB] code4lib.org hosting
On 8/2/07, Ross Singer [EMAIL PROTECTED] wrote: I suppose these are really easy to say but I still haven't seen you asking how you can help or what you can do get this back up since the incident (until now). In fact, nobody has. Until Jeremy's (perfectly reasonable) point that the C4L08 planning is there, nobody has even indicated that this is more than an inconvenience (short of the journal site - but even then, we've come up with alternatives). I thought the site would be back up in some state earlier this week, and like you I've been busy in the meantime. Seeing that it was still down after 10 days led me to want to say something and offer to help. I mean, it's one thing to say that 'we need to have policies and procedures in case of emergency', it's another thing to actually create them and approve them and implement them. I didn't say that. Then there's the fact that it's quite likely that our drupal instance is what was the cause of the break-in in the first place, meaning that would have to be addressed or the mirror site is just as vulnerable. Nobody knows this for sure, so it's only speculation, and doesn't help. The fact that it's quite likely? I guess my point is, I don't see much point in criticizing the all-volunteer effort and donated server space that has gone into code4lib.org now. My goal isn't to criticize (I understand that it might sound like it is). I would just mainly like to see something go back up soon. Being complainy is counterproductive, honestly. If we need to fix the governance issues, then let's do it. But don't assume they'll be taken care of yesterday. I'm not complaining about governance issues. Did I use the word governance? I want to see the site back up and can help if need be. It's impossible to know if I can help get the site back up if I don't know why it isn't back up already. -Dan
Re: [CODE4LIB] code4lib.org hosting
From: Code for Libraries [mailto:[EMAIL PROTECTED] On Behalf Of Ross Singer Sent: 01 August, 2007 23:31 To: CODE4LIB@LISTSERV.ND.EDU Subject: Re: [CODE4LIB] code4lib.org hosting I don't think anything has been 'decided'. We had a meeting, OSU stepped forward, nobody present objected. www.code4lib.org is still 'down'. Has anybody considered talking with the OCLC WebJunction folks about hosting www.code4lib.org? They are already hosting the xml4lib listserve. I do realize we are talking about a domain vs. a listserve, but part of WebJunction's mandate from their Gates grant is community service for libraries. The www.code4lib.org seems to fit within the scope of the public service that they are doing for the xml4lib listserve. So maybe this might be another option to look into. Andy.
Re: [CODE4LIB] code4lib.org hosting
Yes, I have a concern about a university hosting arrangement like what you've described, though it doesn't have anything to do with OSU or you. :) ibiblio is in the business of being a neutral hosting space for sites like this (and, yes, they're at a university, but, their long-lived project there is long-dedicated to this). Special arrangements to host code4lib.org at any particular university through the good graces of one or two people like you are just that - special arrangements, predicated on a generally favorable situation and the kindness of a couple of supportive individuals both in our community - presumably you and rordway - and at OSU (your boss and network/server support staff). If I might pipe up a little... I've lived through the orphaned university hosting scenario for two services (lii.org and the PUBLIB list). To me, the big questions are: * Who can provide the clearest, best-documented relationship (the deliverables question) so it is not all based on handshakes and who knows whom... Dan's special arrangements concern * Which option provides the best service package * Which option has the chance of lasting the longest (moving hurts) * Which option has the clearest, easiest exit strategy (because eventually everybody moves on... it's not a marriage, it's just a protracted date) * Which option gives the community the most ownership of its content * Which option has a proven track record with this kind of relationship (which might also mean, which option has the most at stake for delivering good service to this kind of arrangement) I resisted several special arrangements for lii.org's move from Berkeley SunSITE precisely because I had experience with them. When the person who cares moves on, you can be left twisting in the breeze. (Plus in several cases the suggested arrangements were ridiculous... moving to poorly-maintained Windows servers with known bandwidth problems and suspect security, etc.) Having a special arrangement can be tough. Even when you DO know the people and they have no intention of leaving, they can get very very busy, and their organization always has to come first. I don't have strong feelings about ibiblio versus OSU (without knowing more about either of them), but I see those as the issues. I wonder if code4lib couldn't be just as happy holding an annual virtual bake sale or raffle and buying space at Hurricane Electric or something. Just a thought. What are we talking about in terms of needs? Karen G. Schneider [EMAIL PROTECTED]
Re: [CODE4LIB] code4lib.org hosting
There's too many argument off-shoots so I'll just sum up my overly personal opinions here. * I agree with Ross and Co. about their concerns. When I offered AADL as a possibility I was doing more so for the short term to get a site up and running at least until the conference. Long-term I think would require some binding agreement to make it palatable at any institution, even a university. I think AADL, OSU, etc would all be stop-gaps at the most in the short-term. * I think in the long-run it would make sense to either revisit the idea of non-profit status or find a paid colo host and include the cost either in the yearly conference or by donations. I think this thread shows that there needs to be something resembling governance. Getting someone to admin the box would be another challenge. * I didn't argue that much in channel truthfully because I didn't have an alternative to bring to the table which I thought was workable. It sounds like dchud has more experiences to give input on. * Hosting at a vendor that may be criticized, I think is an obviously not great idea. Which is another concern for any library that offers. This is why I don't believe aadl would not be a long-term solution as stands and I'm weary of others. I think many in the community see it as a source of trustful information and might see sponsorship or hosting as a possible compromise of that trust. * In the end I think this is more a problem with what people think of the community, their responsibilities in the community and the future of code4lib then a simple hosting problem. Democracies seem to involve more work. * The core system was pretty much up to date on anvil, the web apps mostly weren't as can be expected with independent users. Any proposal should probably include details and who is responsible for software upgrades such as the code4lib site and what is expected. I'm game for whatever the community decides. Eby On 8/2/07, Ross Singer [EMAIL PROTECTED] wrote: This much more soundly articulates my concern (I was using university counsel as an example, but anyone in the chain can potentially disrupt this entire community for whatever their reason). Ed and I actually shared this concern (well, I did and Ed was probably idle and wasn't disagreeing). We saw something similar recently: John Blyberg had offered a similar sort of hosting service at AADL. I asked him about what would happen to said service if, on the odd chance, he were to leave. He was rather vague about it, but said it would be the responsibility of his successor. About two weeks later he announced his resignation and there has been nothing about this (that I know of) since. Maybe Ryan Eby has more info here. This isn't a criticism of Blyberg, AADL or good intentions. It's just reality. And I think it illustrates the point perfectly. -Ross.
Re: [CODE4LIB] code4lib.org hosting
On Thu, Aug 02, 2007 at 09:38:00AM -0400, Jonathan Rochkind wrote: Dan convinces me that iBiblio might be better than OSU, in terms of their type of organization. But OSU has made an offer of free supported hosting, under terms that to me seem quite acceptable. Unless iBiblio makes a similar offer under equal or better terms... I still think that an imperfect solution (we all know about what happens when you insist on perfection, right?) is better than the current situation. Dan has in fact articulated quite well what is wrong with the current situation. So Dan and everyone else agrees that it is desirable for the current situation to change. The OSU proposal is the best thing on the table right now for an improvement. If Dan or anyone else wants to spend time on putting together another possibility, that would certainly be welcome. If not, nothing is permanent, we can always change again later (although of course it's a pain and we should not plan on doing such). For once, I agree with J-Ro. Somewhat. :) We don't need to put all of our eggs in one basket. Different parts of the site should be hosted in different places, and the base domain should be fully mirrored. If OSU works right now, let's get it up and running there. When we get it hosted at iBiblio, we can switch the DNS, but leave it running at OSU in case there's a problem with the site at iBiblio. It should be mirrored, with rolling backups, on a couple of other servers as well. If we're just talking about the base web site, it's not that many bytes. It sounds like iBiblio may be our best bet for long-term hosting, but we need to get the site up quickly. How soon could we get something on their servers? To that end, I gathered from Ed's message that the real work involved in getting the site up will be in salvaging what we can from the corrupted database and files. Who will step up to the plate for this? Gabe
Re: [CODE4LIB] code4lib.org hosting
It sounds like iBiblio may be our best bet for long-term hosting, but we need to get the site up quickly. How soon could we get something on their servers? Not that quickily - probably at least a few weeks. It took at least a month - if not more - for them to respond to my application for hosting ArchivesBlogs. Mark A. Matienzo [EMAIL PROTECTED] Assistant Archivist, Niels Bohr Library Archives Center for History of Physics American Institute of Physics 1 Physics Ellipse College Park, MD 20740-3843 USA tel. +1 301.209-3180 - fax +1 301.209-0882 Disclaimer: Opinions in this message are mine alone and do not represent those of the American Institute of Physics, the Society of American Archivists or any other affilates, corporate or individual.
[CODE4LIB] code4lib.org hosting discussion transcript
Okay, no objections, and it's afternoon in Philly, so here it is. I think the following snippet pretty much covers yesterday's discussion. I don't think anyone said anything too incriminating. Please excuse my out-of-place Helen Thomas incrementing. Oh, and, mjgiarlo, I hope you didn't mean for that exclamation point to be real. 2007-08-01T14:57:53 *** jaf changes topic to discussion on hosting code4lib.org 2007-08-01T14:58:03 ksclarke the time has come, eh? 2007-08-01T14:58:09 jaf in a minute, yes :) 2007-08-01T14:58:28 wtd Attention! Attention! Discussion beginning soon. 2007-08-01T14:58:35 jaf ok, the time is upon us 2007-08-01T14:58:36 dbs wtd: Sigh. Check my blog post. 2007-08-01T14:58:37 wtd Everyone load up their channel loggers. 2007-08-01T14:59:00 jaf roll call, please: let's make sure folks are active 2007-08-01T14:59:02 jaf I'm here 2007-08-01T14:59:05 ksclarke here 2007-08-01T14:59:12 rsinger tom servo! 2007-08-01T14:59:12 jbrinley moo 2007-08-01T14:59:14 jrochkind I'm observing. 2007-08-01T14:59:16 rsinger crw! 2007-08-01T14:59:25 wtd I'm here. I have an account on anvil (no root) and host a Rails site on it. 2007-08-01T14:59:33 ksclarke edsu back yet? 2007-08-01T14:59:40 wickr I'm observing 2007-08-01T14:59:43 wtd anvil.lisforge.net, that is, the box that got hacked. 2007-08-01T14:59:49 jbrinley .seen edsu 2007-08-01T14:59:49 zoia jbrinley: edsu was last seen in #code4lib 4 hours, 18 minutes, and 31 seconds ago: edsu like the librarything guys talk, and others 2007-08-01T15:00:09 jaf my local clock says 11:59, so let's wait another minute or so 2007-08-01T15:00:17 ksclarke sounds good 2007-08-01T15:00:20 wickr edsu said he might be able to pop in for a bit, and he might not 2007-08-01T15:00:35 rsinger plus, these are library types-- we need to give the customary 5 minutes 2007-08-01T15:00:38 * rordway is here 2007-08-01T15:00:43 ksclarke rsinger++ 2007-08-01T15:00:56 jaf ok, well, we probably should begin the discussion 2007-08-01T15:00:57 rordway according to my Mac, it's now 12:00 2007-08-01T15:01:06 jaf the proposal on the table, as I understand it, is thus: 2007-08-01T15:01:18 jaf move the production version of code4lib.org over to a server here at OSU 2007-08-01T15:01:25 *** rob_desk has joined #code4lib 2007-08-01T15:01:30 jaf use anvil as a development enivornment 2007-08-01T15:01:41 jaf and set up policies for admin support and access of code4lib.org 2007-08-01T15:01:51 wtd Is that *.code4lib.org? journal, planet, etc? 2007-08-01T15:02:02 jaf wtd: yes, *.code4lib.org 2007-08-01T15:02:06 rsinger hrm 2007-08-01T15:02:12 rsinger dilettantes? 2007-08-01T15:02:15 jaf so, currently we are talking www, planet, and journal 2007-08-01T15:02:16 *** tholbroo has quit IRC 2007-08-01T15:02:29 rsinger jaf: there's more -- svn 2007-08-01T15:02:35 jaf ok, and svn :) 2007-08-01T15:02:37 ksclarke and trac 2007-08-01T15:02:38 jaf and trac 2007-08-01T15:02:39 rsinger trac? 2007-08-01T15:02:40 rsinger yeah 2007-08-01T15:02:46 jrochkind ++ 2007-08-01T15:02:52 wtd There are about, what, ten other more or less production sites hosted on the box? 2007-08-01T15:03:01 jaf wtd: what are those? 2007-08-01T15:03:05 jrochkind Will OSU donate this service? Does this include sysadmin staffing, or just hardware/network, or what? 2007-08-01T15:03:17 ksclarke wtd, what, code4lib things or other people's things? 2007-08-01T15:03:17 rordway [a-zA-Z+].code4lib? :-) 2007-08-01T15:03:38 ksclarke we're only talking code4lib stuff I believe 2007-08-01T15:03:43 wtd Ah, OK. 2007-08-01T15:03:46 jaf jrochkind: we are donating the server space, bandwith, and will support the software running on the box in terms of security and uptime 2007-08-01T15:03:52 jrochkind Awesome. 2007-08-01T15:03:59 wtd So this is a sort of formalization and Oregon State adoption of code4lib.org as an online presence. 2007-08-01T15:04:06 ksclarke osu++ 2007-08-01T15:04:08 jaf but we'd also like some commitment from the community for helping with the general admin of the software 2007-08-01T15:04:12 rsinger hmm 2007-08-01T15:04:17 rsinger i'm still not sure about this 2007-08-01T15:04:18 jaf wtd: no 2007-08-01T15:04:31 ksclarke so how will you manage letting people have the privs for that help, jaf? 2007-08-01T15:04:33 jbrinley jaf: commitment of what sort? 2007-08-01T15:04:37 rsinger 1) my online presence is in the code4lib.org domain 2007-08-01T15:04:49 jaf in other words, we're not going to set policies on / about code4lib.org 2007-08-01T15:04:52 bradl jaf: sounds like you have it handled :) 2007-08-01T15:05:01 jaf we're going to commit to a level of support to assure uptime 2007-08-01T15:05:09 ksclarke yeah, rsinger, yours is the exception (personal in the domain) 2007-08-01T15:05:12 jaf but other than that, it's still the community that controls c4l.org 2007-08-01T15:05:15 rsinger 2) what if, say, osu counsel (or anyone in the chain)
Re: [CODE4LIB] code4lib.org hosting
In case anybody (like me) missed the deface: http://www.zone-h.org/index2.php?option=com_mirrorwrpItemid=43id=6499108 Andrew Nagy wrote: In case I can't make the conversation, I must suggest Bastille - a linux package that does firewalling and IP Masquerading. I have been using it for about 8 years now and have never had a hacked linux box running it. I even had my ISP kill my network connection once because my server was being attacked by thousands of machines and never once got through and the machine never experienced any performance degredation. http://www.bastille-linux.org/ Good luck Andrew -Original Message- From: Code for Libraries [mailto:[EMAIL PROTECTED] On Behalf Of Ed Summers Sent: Friday, July 27, 2007 5:18 PM To: CODE4LIB@listserv.nd.edu Subject: [CODE4LIB] code4lib.org hosting As you may have seen or experienced code4lib.org is down for the count at the moment because of some hackers^w crackers who compromised anvil and defaced various web content and otherwise messed with the operating system. anvil is a machine that several people in the code4lib community run and pay for themselves. Given that code4lib has grown into a serious little gathering, with lots of effort being expended by the likes of Jeremy Frumkin and Brad LaJenuesse to make things happen -- it seems a shame to let this sort of thing happen. We don't have any evidence, but it seems that the entry point was the fact that various software packages weren't kept up to date. Anyhow, this is a long way of inviting you to a discussion Aug 1st @7PM GMT in irc://chat.freenode.net/code4lib to see what steps need to be taken to help prevent this from happening in the future. Specifically we're going to be talking about moving some of the web applications to institutions that are better set up to manage them. If this interests you at all try to attend! //Ed
Re: [CODE4LIB] code4lib.org hosting
Just a reminder, everyone, this conversation is today at 7PM GMT (3PM EDT/Noon PDT) in #code4lib. Hope to see you all there, -Ross. On 7/27/07, Ed Summers [EMAIL PROTECTED] wrote: As you may have seen or experienced code4lib.org is down for the count at the moment because of some hackers^w crackers who compromised anvil and defaced various web content and otherwise messed with the operating system. anvil is a machine that several people in the code4lib community run and pay for themselves. Given that code4lib has grown into a serious little gathering, with lots of effort being expended by the likes of Jeremy Frumkin and Brad LaJenuesse to make things happen -- it seems a shame to let this sort of thing happen. We don't have any evidence, but it seems that the entry point was the fact that various software packages weren't kept up to date. Anyhow, this is a long way of inviting you to a discussion Aug 1st @7PM GMT in irc://chat.freenode.net/code4lib to see what steps need to be taken to help prevent this from happening in the future. Specifically we're going to be talking about moving some of the web applications to institutions that are better set up to manage them. If this interests you at all try to attend! //Ed
Re: [CODE4LIB] code4lib.org hosting
I look forward to the proposal from OSU that should be mailed out to the list shortly. The discussion that just took place in #code4lib got me thinking. As I see it, the issue here has two parts. First, the machine was cracked, and, second, service hasn't been restored following the attack. The code4lib.org site and its various subdomains have served a community with a variety of needs, many of which require command line access and the ability to install programs and services. Maybe some increased restriction as to who has this access and what may be done with it is called for, but even with greater restriction and more vigilant sysadmins it's likely that the machine will get cracked again at some point. While I hope we'll have a more secure box for code4lib in the future, I'm also excited about plans for a system that can bounce back quicker. In addition to local and remote backups, we could use full mirrors ready for a dns switch. Several mirror host machines were even offered in the discussion. Are there other strategies we might employ to make code4lib.org more resilient? On Fri, Jul 27, 2007 at 05:18:06PM -0400, Ed Summers wrote: As you may have seen or experienced code4lib.org is down for the count at the moment because of some hackers^w crackers who compromised anvil and defaced various web content and otherwise messed with the operating system. anvil is a machine that several people in the code4lib community run and pay for themselves. Given that code4lib has grown into a serious little gathering, with lots of effort being expended by the likes of Jeremy Frumkin and Brad LaJenuesse to make things happen -- it seems a shame to let this sort of thing happen. We don't have any evidence, but it seems that the entry point was the fact that various software packages weren't kept up to date. Anyhow, this is a long way of inviting you to a discussion Aug 1st @7PM GMT in irc://chat.freenode.net/code4lib to see what steps need to be taken to help prevent this from happening in the future. Specifically we're going to be talking about moving some of the web applications to institutions that are better set up to manage them. If this interests you at all try to attend! //Ed
Re: [CODE4LIB] code4lib.org hosting
Gabe, I think the OSU proposal addresses your concerns (having people volunteer redundant servers is also a great idea). The machine that was cracked hasn't bounced back quickly because I'm the only one with physical access to it and I've been on vacation. I'm back and waiting now on getting an access pass (which should be assigned to me tomorrow) so that I can get in and swap out the hard drive (with one with a fresh OS)). We have the backups from Anvil though so movement to a new machine at OSU doesn't really need to wait on anvil at this point. Anvil really was never intended to be a production machine and having Code4Lib hosted at OSU where there is a sysadmin attending to it (and policies about access, what can be installed, etc.) seems to me like it will solve the problems we've had in the past. It was fine letting Code4Lib grow a little in the anvil space, but I think the needs of its community have outgrown anvil (and I think this was the general consensus in the channel today). Thanks to OSU for stepping up and giving us a viable alternative! I know we'll have at least two places willing to mirror the Code4Lib site. The more the merrier though! Kevin On 8/1/07, Gabriel Farrell [EMAIL PROTECTED] wrote: I look forward to the proposal from OSU that should be mailed out to the list shortly. The discussion that just took place in #code4lib got me thinking. As I see it, the issue here has two parts. First, the machine was cracked, and, second, service hasn't been restored following the attack. The code4lib.org site and its various subdomains have served a community with a variety of needs, many of which require command line access and the ability to install programs and services. Maybe some increased restriction as to who has this access and what may be done with it is called for, but even with greater restriction and more vigilant sysadmins it's likely that the machine will get cracked again at some point. While I hope we'll have a more secure box for code4lib in the future, I'm also excited about plans for a system that can bounce back quicker. In addition to local and remote backups, we could use full mirrors ready for a dns switch. Several mirror host machines were even offered in the discussion. Are there other strategies we might employ to make code4lib.org more resilient? On Fri, Jul 27, 2007 at 05:18:06PM -0400, Ed Summers wrote: As you may have seen or experienced code4lib.org is down for the count at the moment because of some hackers^w crackers who compromised anvil and defaced various web content and otherwise messed with the operating system. anvil is a machine that several people in the code4lib community run and pay for themselves. Given that code4lib has grown into a serious little gathering, with lots of effort being expended by the likes of Jeremy Frumkin and Brad LaJenuesse to make things happen -- it seems a shame to let this sort of thing happen. We don't have any evidence, but it seems that the entry point was the fact that various software packages weren't kept up to date. Anyhow, this is a long way of inviting you to a discussion Aug 1st @7PM GMT in irc://chat.freenode.net/code4lib to see what steps need to be taken to help prevent this from happening in the future. Specifically we're going to be talking about moving some of the web applications to institutions that are better set up to manage them. If this interests you at all try to attend! //Ed
Re: [CODE4LIB] code4lib.org hosting
It would be helpful if somebody could post a transcript of this discussion. -Dan
Re: [CODE4LIB] code4lib.org hosting
On 8/1/07, D Chudnov [EMAIL PROTECTED] wrote: Okay, I've read a transcript copy that somebody sent me privately. I have a few concerns that I'm going to voice strongly, and I think they represent questions that need to be answered before I'll be comfortable with any particular plan. It doesn't mean I don't love you. Whatever. There was that night after the Kappa Sig social, but you haven't called since. 1. Why isn't the www.code4lib.org site already back up (at minimum)? When the server became unavailable during c4lc 2007, we were able to restore from offsite backups and have an emergency-mode snapshot live and useful within a matter of a few hours at most. It seems this could have been done within a few days if there had been offsite backups available. Eh, what? We have offsite backups, yes. What we don't have are: 1) anywhere to put them back to 2) a plan to proceed once something is back up 3) the resources to dedicate to get code4lib.org running as it was (temporarily, I might add), as in somebody that has the time to do the work (remember, we didn't even have time to keep it up to date in the first place) 4) a hugely pressing need (we're not, after all, in the middle of the eponymous conference) 2. Are there offsite backups of the www.code4lib.org site - its files and database? Yes, and they live on my machine at work. If the hackers^Wcrackers converge on rsinger.library.gatech.edu tonight, we're sunk. 3. The discussion seemed to only involve one proposal. There wasn't a call for any other proposals, and it wasn't clear to me that by missing this meeting (i was at a gathering with several other people with an interest in anvil and other things c4l) that I would miss out on any opportunity to have input. I'd like to propose a different hosting plan. Shouldn't there be a chance for more discussion here? Who's stopping you? We announced a 'town hall meeting', nobody publicly dissented on the mailing list. We're 'discussing' now, you're complaining but not articulating an alternative. You mentioned ibiblio in channel, but until something tangible is offered, the Oregon State offer meets the 'one in hand is worth two in the bush' criteria. 4. Could somebody please post (to this list) an exact statement of what the current proposal is? Supposedly Jeremy Frumkin, Ryan Ordway, Ed Summers and Kevin Clarke will work this out and announce it publicly. 5. Could somebody please post the transcript to the list? This would be useful. Then my reservations about this decision (which may not seem obvious from this email) could go on public record. I understand that some of these questions might seem to be coming a bit late, and I'm sorry to be in a position where my jerkiness is all the worse because of it. But I still think these are questions that need answers. I don't think anything has been 'decided'. We had a meeting, OSU stepped forward, nobody present objected. www.code4lib.org is still 'down'. -Ross.
Re: [CODE4LIB] code4lib.org hosting
On 8/1/07, Ross Singer [EMAIL PROTECTED] wrote: 1. Why isn't the www.code4lib.org site already back up (at minimum)? Eh, what? We have offsite backups, yes. What we don't have are: 1) anywhere to put them back to 2) a plan to proceed once something is back up 3) the resources to dedicate to get code4lib.org running as it was (temporarily, I might add), as in somebody that has the time to do the work (remember, we didn't even have time to keep it up to date in the first place) 4) a hugely pressing need (we're not, after all, in the middle of the eponymous conference) 1) we could have found someplace 2) this is a problem either way 3) this is a problem either way 4) a few days' downtime is one thing - a few weeks' is another. 2. Are there offsite backups of the www.code4lib.org site - its files and database? Yes, and they live on my machine at work. Let's set up a second backup of your backup? I'll take a copy, and add another to s3 for safer-keeping. 3. The discussion seemed to only involve one proposal. There wasn't a call for any other proposals, and it wasn't clear to me that by missing this meeting (i was at a gathering with several other people with an interest in anvil and other things c4l) that I would miss out on any opportunity to have input. I'd like to propose a different hosting plan. Shouldn't there be a chance for more discussion here? Who's stopping you? We announced a 'town hall meeting', nobody publicly dissented on the mailing list. We're 'discussing' now, you're complaining but not articulating an alternative. You mentioned ibiblio in channel, but until something tangible is offered, the Oregon State offer meets the 'one in hand is worth two in the bush' criteria. Nobody is stopping me. I was offline all weekend, and busy otherwise, and it wasn't clear to me whether this was a done deal already. I'm not complaining, I'm just seeking clarification. In a complainy-sounding way. I understand that some of these questions might seem to be coming a bit late, and I'm sorry to be in a position where my jerkiness is all the worse because of it. But I still think these are questions that need answers. I don't think anything has been 'decided'. We had a meeting, OSU stepped forward, nobody present objected. www.code4lib.org is still 'down'. I propose that we move hosting of www.code4lib.org to ibiblio if they'll have us. They've been there for 15(+?) years, they are there for exactly this purpose, and they're not for-profit. I've had good luck hosting things there, and they're liberal about accounts, so long as you don't prove to be an idiot. To support this I'd be happy to sign up for support duties. Separately I can bring up an emergency/temporary backup of the www.code4lib.org site if it is not otherwise possible before the end of the week. I'm in transit all day tomorrow, but let me know by first thing friday -Dan
Re: [CODE4LIB] code4lib.org hosting
On 8/1/07, D Chudnov [EMAIL PROTECTED] wrote: Separately I can bring up an emergency/temporary backup of the www.code4lib.org site if it is not otherwise possible before the end of the week. I'm in transit all day tomorrow, but let me know by first thing friday Shoot, sorry, eager ibook touchpad sent that too soon. What I was going to finish was: let me know by first thing friday a.m. if there's a problem bringing the site back up temporarily and I'll re-load the same backup server instance I had before as soon as I can get a copy of the newer data. -Dan
Re: [CODE4LIB] code4lib.org hosting
In case I can't make the conversation, I must suggest Bastille - a linux package that does firewalling and IP Masquerading. I have been using it for about 8 years now and have never had a hacked linux box running it. I even had my ISP kill my network connection once because my server was being attacked by thousands of machines and never once got through and the machine never experienced any performance degredation. http://www.bastille-linux.org/ Good luck Andrew -Original Message- From: Code for Libraries [mailto:[EMAIL PROTECTED] On Behalf Of Ed Summers Sent: Friday, July 27, 2007 5:18 PM To: CODE4LIB@listserv.nd.edu Subject: [CODE4LIB] code4lib.org hosting As you may have seen or experienced code4lib.org is down for the count at the moment because of some hackers^w crackers who compromised anvil and defaced various web content and otherwise messed with the operating system. anvil is a machine that several people in the code4lib community run and pay for themselves. Given that code4lib has grown into a serious little gathering, with lots of effort being expended by the likes of Jeremy Frumkin and Brad LaJenuesse to make things happen -- it seems a shame to let this sort of thing happen. We don't have any evidence, but it seems that the entry point was the fact that various software packages weren't kept up to date. Anyhow, this is a long way of inviting you to a discussion Aug 1st @7PM GMT in irc://chat.freenode.net/code4lib to see what steps need to be taken to help prevent this from happening in the future. Specifically we're going to be talking about moving some of the web applications to institutions that are better set up to manage them. If this interests you at all try to attend! //Ed
[CODE4LIB] code4lib.org hosting
As you may have seen or experienced code4lib.org is down for the count at the moment because of some hackers^w crackers who compromised anvil and defaced various web content and otherwise messed with the operating system. anvil is a machine that several people in the code4lib community run and pay for themselves. Given that code4lib has grown into a serious little gathering, with lots of effort being expended by the likes of Jeremy Frumkin and Brad LaJenuesse to make things happen -- it seems a shame to let this sort of thing happen. We don't have any evidence, but it seems that the entry point was the fact that various software packages weren't kept up to date. Anyhow, this is a long way of inviting you to a discussion Aug 1st @7PM GMT in irc://chat.freenode.net/code4lib to see what steps need to be taken to help prevent this from happening in the future. Specifically we're going to be talking about moving some of the web applications to institutions that are better set up to manage them. If this interests you at all try to attend! //Ed
