Yeah, the Settings.php was overriden by the latest puppet run. We need to wait for some infra guys to approve my patches and make it permanent: https://review.openstack.org/285669 Disable standard password based auth https://review.openstack.org/285672 Disable mobile frontend
M. On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <[email protected]> wrote: > FYI. Still seeing the mobile view... > > J.P. Maxwell | tipit.net | fibercove.com > On Feb 27, 2016 6:53 AM, "Marton Kiss" <[email protected]> wrote: > >> Yes, applied them manually. Let's wait a few hours, and check for new >> spam content / user accounts. >> >> M. >> JP Maxwell <[email protected]> (időpont: 2016. febr. 27., Szo, 13:50) ezt >> írta: >> >>> Cool. Are these applied? Any indication it has stopped the spam? Should >>> we clear out these non launchpad accounts from the DB? >>> >>> J.P. Maxwell | tipit.net | fibercove.com >>> On Feb 27, 2016 6:47 AM, "Marton Kiss" <[email protected]> wrote: >>> >>>> And the mobile frontend will be disabled permanently with this patch: >>>> https://review.openstack.org/285672 Disable mobile frontend >>>> >>>> M. >>>> >>>> On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss <[email protected]> >>>> wrote: >>>> >>>>> I made some investigation, and it seems to be that the spam pages are >>>>> created by accounts registered with password accounts, and the launchpad >>>>> openid auth is not affected at all. >>>>> >>>>> So the spam script is creating accounts like this: >>>>> mysql> select * from user where user_name = 'CedricJamieson'\G; >>>>> *************************** 1. row *************************** >>>>> user_id: 7494 >>>>> user_name: CedricJamieson >>>>> user_real_name: Cedric Jamieson >>>>> user_password: >>>>> :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY= >>>>> user_newpassword: >>>>> user_newpass_time: NULL >>>>> user_email: [email protected] >>>>> user_touched: 20160227052454 >>>>> user_token: 7c39e44e849fb0e2bfae8790d6cc1379 >>>>> user_email_authenticated: NULL >>>>> user_email_token: be963ac3bd43e70ff2f323063c61e320 >>>>> user_email_token_expires: 20160305052441 >>>>> user_registration: 20160227052441 >>>>> user_editcount: 2 >>>>> user_password_expires: NULL >>>>> >>>>> The user_password field is always filled with a value, meanwhile this >>>>> field of non-infected user accounts with openid logins is empty. >>>>> We have 423 total accounts with passwords: >>>>> mysql> select count(*) from user where user_password != ''; >>>>> +----------+ >>>>> | count(*) | >>>>> +----------+ >>>>> | 423 | >>>>> +----------+ >>>>> 1 row in set (0.00 sec) >>>>> >>>>> Mediawiki logs-in the newly created users without any preliminary >>>>> email confirmation, right after the registration. I disabled the standard >>>>> user login page, as described here: >>>>> https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages >>>>> >>>>> And I made this patch to make it permanent: >>>>> https://review.openstack.org/285669 Disable standard password based >>>>> auth >>>>> >>>>> Just for the record, the last spam user account: >>>>> 7536 | EarthaChester22 >>>>> >>>>> Marton >>>>> >>>>> >>>>> On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I created the following patch, infra cores must approve that: >>>>>> https://review.openstack.org/285641 Add ssh key of JP Maxwell to >>>>>> wiki.o.o >>>>>> >>>>>> Marton >>>>>> >>>>>> On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell <[email protected]> wrote: >>>>>> >>>>>>> Marton has SSH access and applied a patch earlier today. It appears >>>>>>> the spam continues to flow: >>>>>>> >>>>>>> >>>>>>> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen >>>>>>> >>>>>>> Marton let me know if you can look at it some more or Infra if you >>>>>>> want to give me SSH I'll do so as well in the morning (public key >>>>>>> attached). >>>>>>> >>>>>>> >>>>>>> >>>>>>> ssh-rsa >>>>>>> AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q== >>>>>>> [email protected] >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> J.P. Maxwell / tipit.net <http://www.tipit.net> >>>>>>> >>>>>>> >>>>>>> On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Super thankful for all the folks that have jumped in over the last >>>>>>>> couple of days to help with the puppetization, etc... I just feel like >>>>>>>> we're taking a very wrong approach here. >>>>>>>> >>>>>>>> Paul Belanger wrote: >>>>>>>> >>>>>>>> Right, and I don't have an issue with that approach. Based on the >>>>>>>> work we did >>>>>>>> yesterday, anybody can do that via our workflow. Please submit a patch >>>>>>>> to >>>>>>>> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC. >>>>>>>> >>>>>>>> What I'm proposing is the workflow is really meant for software, >>>>>>>> not for web applications. It's tedious and time consuming when what's >>>>>>>> needed here is a set of tests on the server. Submitting a patch, >>>>>>>> waiting >>>>>>>> for a +1, then getting on IRC to find someone with access (and time) to >>>>>>>> paste the logs is a pretty time consuming process for what should be a >>>>>>>> series of rapid-fire changes/fixes on the server. Especially when we're >>>>>>>> dealign with an active attack. >>>>>>>> >>>>>>>> >>>>>>>> We can then have somebody look at the logs. I think it is more about >>>>>>>> scheduling >>>>>>>> the task since more infra-root as travling back from the mid-cycle >>>>>>>> last night >>>>>>>> and today. >>>>>>>> >>>>>>>> Right, this is my point. This has been going on for 3 weeks (or >>>>>>>> more). Tom Fifeldt was asking for help without response. And here we >>>>>>>> are >>>>>>>> through another week and no closer to stemming the flow. >>>>>>>> >>>>>>>> I'm fully aware what I'm proposing goes against what Infra and the >>>>>>>> OpenStack workflow is all about, but I'd ask you all to look at this >>>>>>>> from a >>>>>>>> web development perspective instead of a software development >>>>>>>> perspective. >>>>>>>> >>>>>>>> Jimmy >>>>>>>> >>>>>>>> >>>>>>>> Last email from me, just on a plane. Will follow up when I land. >>>>>>>> >>>>>>>> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki >>>>>>>> >>>>>>>> J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com >>>>>>>> [http://www.fibercove.com] >>>>>>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger >>>>>>>> <[email protected]> <[email protected]> >>>>>>>> wrote: >>>>>>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote: >>>>>>>> >>>>>>>> Given the state of the wiki a the moment, I think taking the quickest >>>>>>>> path >>>>>>>> to get it fixed would be prudent. Is there a way we can get JP root >>>>>>>> access >>>>>>>> to this server, even temporarily? We get 25% of our website traffic (2 >>>>>>>> million visitors) to the wiki. I realize we're all after the same >>>>>>>> thing, >>>>>>>> >>>>>>>> but >>>>>>>> >>>>>>>> spammers are not going to hit the dev environment, so there's really no >>>>>>>> >>>>>>>> way >>>>>>>> >>>>>>>> to tell if teh problem is fixed without actually working directly on >>>>>>>> the >>>>>>>> production machine. This should be a 30 minute fix. >>>>>>>> >>>>>>>> >>>>>>>> I am still unclear what the 30min fix is. If really 30mins, then it >>>>>>>> shouldn't be >>>>>>>> hard to get the fix into our workflow. Could somebody please elaborate. >>>>>>>> >>>>>>>> If we are talking about deploying new versions of php or mediawiki >>>>>>>> manually, >>>>>>>> I >>>>>>>> not be in-favor of this. To me, while the attack sucks, we should be >>>>>>>> working >>>>>>>> on >>>>>>>> 2 fronts. Getting the help needed to mitigate the attack, then adding >>>>>>>> the >>>>>>>> changes into -infra workflow in parallel. >>>>>>>> >>>>>>>> >>>>>>>> I realize there is a lot of risk in giving ssh access to infra >>>>>>>> machines, >>>>>>>> >>>>>>>> but >>>>>>>> >>>>>>>> I think it's worth taking a look at either putting this machine in a >>>>>>>> place >>>>>>>> where a different level of admin could access it without giving away >>>>>>>> the >>>>>>>> keys to the entire OpenStack infrastructure or figuring out a way to >>>>>>>> set >>>>>>>> >>>>>>>> up >>>>>>>> >>>>>>>> credentials with varying levels of access. >>>>>>>> >>>>>>>> >>>>>>>> As a note, all the work I've been doing to help with the attack hasn't >>>>>>>> require >>>>>>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our >>>>>>>> configuration safely. I'd rather take some time to see what the fixes >>>>>>>> are, >>>>>>>> having infra-root apply changes, then move them into puppet. >>>>>>>> >>>>>>>> It also has been discussed to simply disable write access to the wiki >>>>>>>> if we >>>>>>>> really want spamming to stop, obviously that will affect normal usage. >>>>>>>> >>>>>>>> >>>>>>>> Jimmy >>>>>>>> >>>>>>>> Paul Belanger wrote: >>>>>>>> >>>>>>>> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote: >>>>>>>> >>>>>>>> But if you wanted to upgrade everything, remove the mobile view >>>>>>>> >>>>>>>> extension, >>>>>>>> >>>>>>>> test in a dev/staging environment then deploy to production fingers >>>>>>>> crossed, I think that would be a valid approach as well. >>>>>>>> >>>>>>>> >>>>>>>> Current review up[1]. I'll launch a node tonight / tomorrow locally to >>>>>>>> >>>>>>>> see >>>>>>>> how >>>>>>>> >>>>>>>> puppet reacts. I suspect there will be some issues. >>>>>>>> >>>>>>>> If infra-roots are fine with this approach, we can use that box to test >>>>>>>> >>>>>>>> against. >>>>>>>> >>>>>>>> [1] https://review.openstack.org/#/c/285405/ >>>>>>>> >>>>>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>>>>> On Feb 26, 2016 10:08 AM, "JP Maxwell"<[email protected]> >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Plus one except in this case it is much easier to know if our efforts >>>>>>>> >>>>>>>> are >>>>>>>> >>>>>>>> working on production because the spam either stops or not. >>>>>>>> >>>>>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>>>>> On Feb 26, 2016 9:48 AM, "Paul Belanger"<[email protected]> >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote: >>>>>>>> >>>>>>>> I really think you might consider the option that there is a >>>>>>>> >>>>>>>> vulnerability >>>>>>>> >>>>>>>> in one of the extensions. If that is the case black listing IPs will >>>>>>>> >>>>>>>> be >>>>>>>> >>>>>>>> an >>>>>>>> >>>>>>>> ongoing wild goose chase. >>>>>>>> >>>>>>>> I think this would be easily proven or disproven by making the questy >>>>>>>> question impossible and see if the spam continues. >>>>>>>> >>>>>>>> >>>>>>>> We'll have to let an infra-root make that call. Since nobody would be >>>>>>>> able to >>>>>>>> use the wiki. Honestly, I'd rather spend the time standing up a mirror >>>>>>>> >>>>>>>> dev >>>>>>>> >>>>>>>> instance for us to work on, rather then production. >>>>>>>> >>>>>>>> >>>>>>>> J.P. Maxwell | tipit.net | fibercove.com >>>>>>>> On Feb 26, 2016 9:12 AM, "Paul Belanger"<[email protected]> >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote: >>>>>>>> >>>>>>>> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<[email protected]> >>>>>>>> <[email protected]> >>>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote: >>>>>>>> >>>>>>>> Please be aware that you can now create accounts under the mobile >>>>>>>> view in the wiki native user table. I just created an account for >>>>>>>> JpMaxMan. Not sure if this matters but wanted to make sure you >>>>>>>> were aware. >>>>>>>> >>>>>>>> Oh, yes I think having a random garbage question/answer was in >>>>>>>> >>>>>>>> fact >>>>>>>> >>>>>>>> previously preventing account creation under the mobile view. We >>>>>>>> probably need a way to disable mobile view account creation as it >>>>>>>> bypasses OpenID authentication entirely. >>>>>>>> >>>>>>>> So that's what it was doing! We'll have to tackle the mobile view >>>>>>>> >>>>>>>> issue. >>>>>>>> >>>>>>>> Otherwise, quick update here: >>>>>>>> >>>>>>>> The captcha didn't appear to help stem the spam tide. We'll want to >>>>>>>> explore and start implementing some of the other solutions. >>>>>>>> >>>>>>>> I did some database poking around today and it does seem like all >>>>>>>> >>>>>>>> the >>>>>>>> >>>>>>>> users do have launchpad accounts and email addresses. >>>>>>>> >>>>>>>> >>>>>>>> So, I have a few hours before jumping on my plane and checked into >>>>>>>> >>>>>>>> this. >>>>>>>> >>>>>>>> We are >>>>>>>> using QuestyCaptcha which according to docs, should almost be >>>>>>>> >>>>>>>> impossible >>>>>>>> >>>>>>>> for >>>>>>>> spammers to by pass in an automated fashion. So, either our captcha >>>>>>>> >>>>>>>> is too >>>>>>>> >>>>>>>> easy, or we didn't set it up properly. I don't have SSH on wiki.o.o >>>>>>>> >>>>>>>> so >>>>>>>> >>>>>>>> others >>>>>>>> will have to check logs. I did test new pages and edits, and was >>>>>>>> >>>>>>>> promoted >>>>>>>> >>>>>>>> by >>>>>>>> captcha. >>>>>>>> >>>>>>>> As a next step, we might need to add additional apache2 >>>>>>>> >>>>>>>> configuration >>>>>>>> >>>>>>>> to >>>>>>>> >>>>>>>> blacklist IPs. I am reading up on that now. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Elizabeth Krumbach Joseph || Lyz || pleia2 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-Infra mailing [email protected] >>>>>>>> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-Infra mailing [email protected] >>>>>>>> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-Infra mailing >>>>>>>> [email protected]http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-Infra mailing >>>>>>>> [email protected]http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-Infra mailing >>>>>>>> [email protected]http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-Infra mailing list >>>>>>> [email protected] >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra >>>>>>> >>>>>>
_______________________________________________ OpenStack-Infra mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
