Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
One last self-reply for future readers who may judge Eli more by this one interaction than his body of work, let me also add that Eli has been a huge part of whatever success we've had with Racket. He positive influence cannot be overstated. Sorry again for being a jerk, Eli. Robby On Sun, Sep 22, 2013 at 5:58 PM, Robby Findler wrote: > Eli: I'm sorry. I (obviously) didn't mean to send this message publicly > and it was also definitely sent in frustration (in lots of directions, not > only yours). > > My apologies. > > Robby > > > > On Sun, Sep 22, 2013 at 5:44 PM, Robby Findler < > ro...@eecs.northwestern.edu> wrote: > >> It is like he is trying to justify his existence while he is on his way >> out the door which seems strange. IIUC, we cannot afford $80/90k or >> whatever it was he costs anyway. >> >> So I have no idea what to think about his message. >> >> Robby >> >> >> On Sun, Sep 22, 2013 at 5:34 PM, Jay McCarthy wrote: >> >>> Next time, feel free to follow the directions on >>> internal.racket-lang.org. Now that you've turn off its access, rather >>> than just logging in and killing it, I can't test and see what the >>> underlying problem was. Let me know when you have turn traffic back >>> on. >>> >>> Jay >>> >>> On Sun, Sep 22, 2013 at 3:26 PM, Eli Barzilay wrote: >>> > (Note that instead of the apache rule I now switched to a firewall >>> > rule, so it won't even get 403 responses now.) >>> > >>> > >>> > 40 minutes ago, Eli Barzilay wrote: >>> >> Update: bringing it down for a few minutes didn't help, and the >>> >> offending process continues its merciless traffic. I've added a >>> >> temporary rule that effectively blacklists planet access from that IP >>> >> address. (Apologies in case that's a shared machine.) All I see now, >>> >> are failed attempts to get "/servlets/pkg-info.ss" (which are answered >>> >> with a 403 to that IP). >>> >> >>> >> Can someone at BYU look into this? >>> > >>> > -- >>> > ((lambda (x) (x x)) (lambda (x) (x x))) Eli >>> Barzilay: >>> > http://barzilay.org/ Maze is >>> Life! >>> > _ >>> > Racket Developers list: >>> > http://lists.racket-lang.org/dev >>> >>> >>> >>> -- >>> Jay McCarthy >>> Assistant Professor / Brigham Young University >>> http://faculty.cs.byu.edu/~jay >>> >>> "The glory of God is Intelligence" - D&C 93 >>> _ >>> Racket Developers list: >>> http://lists.racket-lang.org/dev >>> >> >> > _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
Just now, Jay McCarthy wrote: > On Sun, Sep 22, 2013 at 6:53 PM, Eli Barzilay wrote: > > > > In any case, if it is the package server through some other > > machine, then it's best to change it so it comes from the actual > > server. > > I don't know what's going on with that. It's in a VM, so maybe > something is fishy when traffic leaves it versus when it comes to > it? Ooh, that's pretty bad for a server. Having an IP address that doesn't resolve back to the IP name is nothing new these days, but having traffic from the server come via a different IP address is really not a good idea. Think about dealing with some kind of an external service, who would need to be aware of your traffic: having it come from a different IP address is something that would make it very hard. It would be a good idea to ask the people who manage that if it's possible to get the expected behavior. (FWIW, it might be some result of a firewall or something like that too. In NEU, our public machines are all in a DMZ network so they're not affected by such firewalling. (But it does mean dealing with a public machine -- for example, dealing with ssh dictionary attacks, not having some kind of expected weaknesses exposed like PHP and similar junkware, etc.)) > It is supposed to do it weekly. I just turned it back on and did not > get an error, so I'm not sure what the problem was. (The 403 errors > totally filled the log, so I couldn't tell what the problem was > earlier in the day.) So, I'm not sure what the problem was. I can tell you exactly when it happend -- the flood started with this entry: 128.187.97.22 - - [21/Sep/2013:22:10:10 -0400] "GET /servlets/pkg-info.ss HTTP/1.1" 200 5650 "-" "-" This was the first entry from that IP address for the whole week, so it was probably the weekly run which then went bad. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
On Sun, Sep 22, 2013 at 6:53 PM, Eli Barzilay wrote: > A few minutes ago, Jay McCarthy wrote: >> >> In retrospect, I guess it's not so obvious that the package server >> contacts the old server regularly to build the compatibility version >> packages. > > Is this the package server?? The IP I have for that is > 128.187.105.226, which is different from the IP that caused the > traffic. This is why I couldn't guess what causes the traffic, and > guessed some rogue experiment in indexing on some test machine. > > In any case, if it is the package server through some other machine, > then it's best to change it so it comes from the actual server. I don't know what's going on with that. It's in a VM, so maybe something is fishy when traffic leaves it versus when it comes to it? >> > It's back on now. >> >> Thanks... it looks like I'm still getting 403s though. > > Ah, sorry -- I forgot to remove the apache rule too. Should be > working now. Yes, thanks. > Also, since it's scanning the planet packages (at least looks like > that), and those really don't change that often, then it'll be much > better to do this scan much more infrequently -- like once every hour > or so rather than once every two seconds... It is supposed to do it weekly. I just turned it back on and did not get an error, so I'm not sure what the problem was. (The 403 errors totally filled the log, so I couldn't tell what the problem was earlier in the day.) So, I'm not sure what the problem was. Jay > > -- > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: > http://barzilay.org/ Maze is Life! -- Jay McCarthy Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay "The glory of God is Intelligence" - D&C 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
A few minutes ago, Jay McCarthy wrote: > > In retrospect, I guess it's not so obvious that the package server > contacts the old server regularly to build the compatibility version > packages. Is this the package server?? The IP I have for that is 128.187.105.226, which is different from the IP that caused the traffic. This is why I couldn't guess what causes the traffic, and guessed some rogue experiment in indexing on some test machine. In any case, if it is the package server through some other machine, then it's best to change it so it comes from the actual server. > > It's back on now. > > Thanks... it looks like I'm still getting 403s though. Ah, sorry -- I forgot to remove the apache rule too. Should be working now. Also, since it's scanning the planet packages (at least looks like that), and those really don't change that often, then it'll be much better to do this scan much more infrequently -- like once every hour or so rather than once every two seconds... -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
On Sun, Sep 22, 2013 at 5:31 PM, Eli Barzilay wrote: > 50 minutes ago, Jay McCarthy wrote: >> Next time, feel free to follow the directions on >> internal.racket-lang.org. > > I have no practical way to know whether it's actually one of your > machines. (I did check that it's not an IP that is in our DNS.) > > >> Now that you've turn off its access, rather than just logging in and >> killing it, > > Nor do I know what "it" is that should be killed. (And I will > certainly not going to ssh into your account and sniff around.) In retrospect, I guess it's not so obvious that the package server contacts the old server regularly to build the compatibility version packages. >> I can't test and see what the underlying problem was. Let me know >> when you have turn traffic back on. > > It's back on now. Thanks... it looks like I'm still getting 403s though. Jay -- Jay McCarthy Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay "The glory of God is Intelligence" - D&C 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
Eli: Apologies for this exchange. Someone who volunteers his time to monitor our machines and fix them up doesn't deserve any kind of yelling and other crap that showed up in this thread. Your help is very much appreciated not to speak of the tons of work from the past. Again, apologies from top-flight management -- Matthias On Sep 22, 2013, at 7:31 PM, Eli Barzilay wrote: > 50 minutes ago, Jay McCarthy wrote: >> Next time, feel free to follow the directions on >> internal.racket-lang.org. > > I have no practical way to know whether it's actually one of your > machines. (I did check that it's not an IP that is in our DNS.) > > >> Now that you've turn off its access, rather than just logging in and >> killing it, > > Nor do I know what "it" is that should be killed. (And I will > certainly not going to ssh into your account and sniff around.) > > >> I can't test and see what the underlying problem was. Let me know >> when you have turn traffic back on. > > It's back on now. > > -- > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: >http://barzilay.org/ Maze is Life! > _ > Racket Developers list: > http://lists.racket-lang.org/dev _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
50 minutes ago, Jay McCarthy wrote: > Next time, feel free to follow the directions on > internal.racket-lang.org. I have no practical way to know whether it's actually one of your machines. (I did check that it's not an IP that is in our DNS.) > Now that you've turn off its access, rather than just logging in and > killing it, Nor do I know what "it" is that should be killed. (And I will certainly not going to ssh into your account and sniff around.) > I can't test and see what the underlying problem was. Let me know > when you have turn traffic back on. It's back on now. -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
Eli: I'm sorry. I (obviously) didn't mean to send this message publicly and it was also definitely sent in frustration (in lots of directions, not only yours). My apologies. Robby On Sun, Sep 22, 2013 at 5:44 PM, Robby Findler wrote: > It is like he is trying to justify his existence while he is on his way > out the door which seems strange. IIUC, we cannot afford $80/90k or > whatever it was he costs anyway. > > So I have no idea what to think about his message. > > Robby > > > On Sun, Sep 22, 2013 at 5:34 PM, Jay McCarthy wrote: > >> Next time, feel free to follow the directions on >> internal.racket-lang.org. Now that you've turn off its access, rather >> than just logging in and killing it, I can't test and see what the >> underlying problem was. Let me know when you have turn traffic back >> on. >> >> Jay >> >> On Sun, Sep 22, 2013 at 3:26 PM, Eli Barzilay wrote: >> > (Note that instead of the apache rule I now switched to a firewall >> > rule, so it won't even get 403 responses now.) >> > >> > >> > 40 minutes ago, Eli Barzilay wrote: >> >> Update: bringing it down for a few minutes didn't help, and the >> >> offending process continues its merciless traffic. I've added a >> >> temporary rule that effectively blacklists planet access from that IP >> >> address. (Apologies in case that's a shared machine.) All I see now, >> >> are failed attempts to get "/servlets/pkg-info.ss" (which are answered >> >> with a 403 to that IP). >> >> >> >> Can someone at BYU look into this? >> > >> > -- >> > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: >> > http://barzilay.org/ Maze is >> Life! >> > _ >> > Racket Developers list: >> > http://lists.racket-lang.org/dev >> >> >> >> -- >> Jay McCarthy >> Assistant Professor / Brigham Young University >> http://faculty.cs.byu.edu/~jay >> >> "The glory of God is Intelligence" - D&C 93 >> _ >> Racket Developers list: >> http://lists.racket-lang.org/dev >> > > _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
It is like he is trying to justify his existence while he is on his way out the door which seems strange. IIUC, we cannot afford $80/90k or whatever it was he costs anyway. So I have no idea what to think about his message. Robby On Sun, Sep 22, 2013 at 5:34 PM, Jay McCarthy wrote: > Next time, feel free to follow the directions on > internal.racket-lang.org. Now that you've turn off its access, rather > than just logging in and killing it, I can't test and see what the > underlying problem was. Let me know when you have turn traffic back > on. > > Jay > > On Sun, Sep 22, 2013 at 3:26 PM, Eli Barzilay wrote: > > (Note that instead of the apache rule I now switched to a firewall > > rule, so it won't even get 403 responses now.) > > > > > > 40 minutes ago, Eli Barzilay wrote: > >> Update: bringing it down for a few minutes didn't help, and the > >> offending process continues its merciless traffic. I've added a > >> temporary rule that effectively blacklists planet access from that IP > >> address. (Apologies in case that's a shared machine.) All I see now, > >> are failed attempts to get "/servlets/pkg-info.ss" (which are answered > >> with a 403 to that IP). > >> > >> Can someone at BYU look into this? > > > > -- > > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: > > http://barzilay.org/ Maze is Life! > > _ > > Racket Developers list: > > http://lists.racket-lang.org/dev > > > > -- > Jay McCarthy > Assistant Professor / Brigham Young University > http://faculty.cs.byu.edu/~jay > > "The glory of God is Intelligence" - D&C 93 > _ > Racket Developers list: > http://lists.racket-lang.org/dev > _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
Next time, feel free to follow the directions on internal.racket-lang.org. Now that you've turn off its access, rather than just logging in and killing it, I can't test and see what the underlying problem was. Let me know when you have turn traffic back on. Jay On Sun, Sep 22, 2013 at 3:26 PM, Eli Barzilay wrote: > (Note that instead of the apache rule I now switched to a firewall > rule, so it won't even get 403 responses now.) > > > 40 minutes ago, Eli Barzilay wrote: >> Update: bringing it down for a few minutes didn't help, and the >> offending process continues its merciless traffic. I've added a >> temporary rule that effectively blacklists planet access from that IP >> address. (Apologies in case that's a shared machine.) All I see now, >> are failed attempts to get "/servlets/pkg-info.ss" (which are answered >> with a 403 to that IP). >> >> Can someone at BYU look into this? > > -- > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: > http://barzilay.org/ Maze is Life! > _ > Racket Developers list: > http://lists.racket-lang.org/dev -- Jay McCarthy Assistant Professor / Brigham Young University http://faculty.cs.byu.edu/~jay "The glory of God is Intelligence" - D&C 93 _ Racket Developers list: http://lists.racket-lang.org/dev
Re: [racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
(Note that instead of the apache rule I now switched to a firewall rule, so it won't even get 403 responses now.) 40 minutes ago, Eli Barzilay wrote: > Update: bringing it down for a few minutes didn't help, and the > offending process continues its merciless traffic. I've added a > temporary rule that effectively blacklists planet access from that IP > address. (Apologies in case that's a shared machine.) All I see now, > are failed attempts to get "/servlets/pkg-info.ss" (which are answered > with a 403 to that IP). > > Can someone at BYU look into this? -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
[racket-dev] Pinging BYU people!! (was: DOS attack on planet?)
Update: bringing it down for a few minutes didn't help, and the offending process continues its merciless traffic. I've added a temporary rule that effectively blacklists planet access from that IP address. (Apologies in case that's a shared machine.) All I see now, are failed attempts to get "/servlets/pkg-info.ss" (which are answered with a 403 to that IP). Can someone at BYU look into this? 20 minutes ago, Eli Barzilay wrote: > I just looked into that, and it seems that there's something bad going > on with some machine at BYU which started yesterday. (Ping: Jay.) > > The offending traffic comes from "fltr5.byu.edu", at a very high rate. > The new log file for the week had started at 2013-09-22 03:40 local > time (about 12.5 hours ago) with 92000 queries for this period, and > 85% of this traffic (about 78k, about a 100 hits per second) is coming > from this BYU IP. Looking back, it seems that it's something recent > that had started just yesterday, so whatever it is, it's new. Most of > the traffic is basically a repeating loop of these 8 lines, shown below. > > (I will restart the server now, in an attempt to get whatever it is > that causes this mess to crash.) -- ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: http://barzilay.org/ Maze is Life! _ Racket Developers list: http://lists.racket-lang.org/dev
