Re: Infineon firmware security issues
Le mardi 17 octobre 2017 à 13:33 +0300, Eyal Edri a écrit : > Thanks, > > So if I have an old YubiKey ( 2.43 ) I shouldn't be affected right? > only V4 > is ? That's what the post on yubico.com seems to imply. We do not know what chipset is used in the key, so I can't give a educated guess. But I hear people using yubikey neo weren't affected. Now, only the CCID function is problematic, and only if you did generate the ssh key on the chip (e.g., followed official doc on https ://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html and used "yubico-piv-tool -s 9a -a generate -o public.pem" ) If you imported the key, then that should be ok. If you use the yubikey for non smartcard use (e.g. U2F, 2FA for RH VPN or similar system ), that's ok too. > On Tue, Oct 17, 2017 at 12:56 PM, Marc Dequènes (Duck) > <d...@redhat.com> > wrote: > > > Quack, > > > > So the news (thanks Misc for the alert): > > > > https://www.infineon.com/cms/en/product/promopages/rsa- > > update/rsa-background > > > > This affects Yubikeys and other hardware: > > https://www.yubico.com/support/security-advisories/ysa-2017-01/ > > > > There's a nice tool to test if a key is vulnerable: > > https://github.com/crocs-muni/roca > > > > I tested keys in the oVirt Puppet repository and none are affected. > > > > You may check your other keys and ensure keys are checked in other > > projects. > > > > \_o< > > > > > > ___ > > Infra mailing list > > Infra@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > > ___ > Infra mailing list > Infra@ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Infineon firmware security issues
Le mardi 17 octobre 2017 à 13:36 +0300, Eyal Edri a écrit : > On Tue, Oct 17, 2017 at 1:31 PM, Michael Scherer <msche...@redhat.com > > > wrote: > > > Le mardi 17 octobre 2017 à 18:56 +0900, Marc Dequènes (Duck) a > > écrit : > > > Quack, > > > > > > So the news (thanks Misc for the alert): > > > > > > https://www.infineon.com/cms/en/product/promopages/rsa-update/rsa > > > -bac > > > kground > > > > > > This affects Yubikeys and other hardware: > > > https://www.yubico.com/support/security-advisories/ysa-2017-01/ > > > > > > There's a nice tool to test if a key is vulnerable: > > > https://github.com/crocs-muni/roca > > > > > > I tested keys in the oVirt Puppet repository and none are > > > affected. > > > > > > You may check your other keys and ensure keys are checked in > > > other > > > projects. > > > > Ideally, if someone could verify the key in Gerrit, it would be > > helpful. I removed mine, but I suspect i am not the only one who > > tried > > to follow best practices :) > > > > If you run the tool locally on your .ssh/ dir, it should include > already > the public key you have on Gerrit no? Well, I know my key is vulnerable, got notified by Fedora and Github. But I just do not know where I used it exactly, because I have account everywhere, and that's likely that I may forget it in some place. > We'll need to check if its possible to run that tool on Gerrit and if > the > keys are even stored on the fs and not inside the Gerrit DB. If they are in the DB, we can extract it with a sql request ILMHO. I plan to look at Gluster's gerrit instance once I finish my own cleanup and key generation, which is a rather tedious task (cause I also found out that my backup key is not working anymore for a unknown reason). -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Infineon firmware security issues
Le mardi 17 octobre 2017 à 18:56 +0900, Marc Dequènes (Duck) a écrit : > Quack, > > So the news (thanks Misc for the alert): > > https://www.infineon.com/cms/en/product/promopages/rsa-update/rsa-bac > kground > > This affects Yubikeys and other hardware: > https://www.yubico.com/support/security-advisories/ysa-2017-01/ > > There's a nice tool to test if a key is vulnerable: > https://github.com/crocs-muni/roca > > I tested keys in the oVirt Puppet repository and none are affected. > > You may check your other keys and ensure keys are checked in other > projects. Ideally, if someone could verify the key in Gerrit, it would be helpful. I removed mine, but I suspect i am not the only one who tried to follow best practices :) Debian, Github and Fedora did sent alert to people affected, and I am in the process of changing my key from the 50 to 60 place where I used it and I assume most affected people will be aware somehow, but automated removal from vulnerable systems would surely help. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Fwd: Undeliverable: confirm 286441e8a96ed0284e7691c2ef4d59dee8c9cbfc
Le mercredi 09 août 2017 à 21:27 +0300, Barak Korren a écrit : > On 9 August 2017 at 21:21, Michael Scherer <m...@redhat.com> wrote: > > > > The problem is that the ssh port is not open. > > ~ $ sudo nmap -sS lists.ovirt.org > > > > Starting Nmap 6.40 ( http://nmap.org ) at 2017-08-09 20:14 CEST > > Nmap scan report for lists.ovirt.org (66.187.230.42) > > Host is up (0.18s latency). > > rDNS record for 66.187.230.42: lists.phx.ovirt.org > > Not shown: 996 filtered ports > > PORTSTATE SERVICE > > 25/tcp open smtp > > 80/tcp open http > > 443/tcp open https > > 993/tcp open imaps > > > > Nmap done: 1 IP address (1 host up) scanned in 15.44 seconds > > > > Not open from anywhere I try in fact. > > > > Yeah we have OpenVPN setup to grant access to PHX servers if you're > not behind the Red Hat firewall. It also fail from inside the firewall using my system in the Paris office. > Evgheny can setup access for you. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Fwd: Undeliverable: confirm 286441e8a96ed0284e7691c2ef4d59dee8c9cbfc
Le mercredi 09 août 2017 à 21:06 +0300, Eyal Edri a écrit : > Hi misc, > ssh to lists is done via public key access, > You need to add your ssh public key to the infra-puppet repo [1] and we > need to update it also in foreman iirc. The key is already there: https://gerrit.ovirt.org/gitweb?p=infra-puppet.git;a=blob;f=site/ovirt_infra/manifests/user/misc.pp;h=b9ab6146f590eea71f225b5ffe0a0f61c8e163c9;hb=42f86bc97dad1910490cdc57c42861fcd66b1970 The problem is that the ssh port is not open. ~ $ sudo nmap -sS lists.ovirt.org Starting Nmap 6.40 ( http://nmap.org ) at 2017-08-09 20:14 CEST Nmap scan report for lists.ovirt.org (66.187.230.42) Host is up (0.18s latency). rDNS record for 66.187.230.42: lists.phx.ovirt.org Not shown: 996 filtered ports PORTSTATE SERVICE 25/tcp open smtp 80/tcp open http 443/tcp open https 993/tcp open imaps Nmap done: 1 IP address (1 host up) scanned in 15.44 seconds Not open from anywhere I try in fact. > I can change it if you'll tell me what do. > > [1] duck's key patch: > https://gerrit.ovirt.org/gitweb?p=infra-puppet.git;a=blob;f=site/ovirt_infra/manifests/user/duck.pp;h=c63ab9151726450419b29fa8f9933d5759e8c4a2;hb=42f86bc97dad1910490cdc57c42861fcd66b1970 > > > On Wed, Aug 9, 2017 at 7:39 PM, Michael Scherer <m...@redhat.com> wrote: > > > Le mardi 08 août 2017 à 12:49 -0700, Karsten Wade a écrit : > > > Misc & Duck: > > > > > > This is what has been flowing sporadically for the last few days (~590 > > > messages so far). > > > > > > It appears to be a full mailbox and continuous subscription requests by > > > a small set of possibly autogenerated email accounts? > > > > > > squ...@live.com > > > sqoon...@live.com > > > squoon+uj...@live.com > > > > > > Is there anything we can do other than blacklisting an entire domain? > > > Maybe tell mailman to auto-reject anything from sq.*@live.com? > > > > It can be done also on postfix directly. > > > > But I can't seems to connect to the list server with ssh, and if there > > is a specific bastion, I can't find it in the wiki or on read the doc, > > or on looking on the ansible infra repository. > > > > > > > - Karsten > > > > > > Forwarded Message > > > Subject: Undeliverable: confirm 286441e8a96ed0284e7691c2ef4d59dee8c9cbfc > > > Date: Tue, 8 Aug 2017 19:39:52 + > > > From: postmas...@outlook.com > > > To: mailman-boun...@ovirt.org > > > > > > Delivery has failed to these recipients or groups: > > > > > > squoon+uj...@live.com<mailto:squoon%2buj...@live.com> > > > The recipient's mailbox is full and can't accept messages now. Please > > > try resending your message later, or contact the recipient directly. > > > > > > > > > > > > > > > > > > > > > > > > > > > Diagnostic information for administrators: > > > > > > Generating server: SN1NAM04HT150.mail.protection.outlook.com > > > > > > squoon+uj...@live.com > > > Remote Server returned '554 5.2.2 mailbox full; > > > STOREDRV.Deliver.Exception:QuotaExceededException. > > MapiExceptionShutoffQuotaExceeded; > > > Failed to process message due to a permanent exception with message > > > Cannot get ID from name. 16.55847:5C06, > > > 17.43559:8C00, > > > 20.52176:010F28854010F103, 20.50032:010F288581171031, > > > 0.35180:, 255.23226:, 255.27962:5600, > > > 255.17082:B904, 0.16993:1400, 4.21921:B904, > > > 255.31418:80030400, 0.22753:80030400, 255.21817:B904, > > > 0.37224:, 4.40808:DD04, 0.24529:382D3763, 4.18385:DD04, > > > 0.36864:30303030, 4.37120:DD04 [Stage: CreateMessage]' > > > > > > Original message headers: > > > > > > Received: from SN1NAM04FT057.eop-NAM04.prod.protection.outlook.com > > > (10.152.88.58) by SN1NAM04HT150.eop-NAM04.prod.protection.outlook.com > > > (10.152.89.108) with Microsoft SMTP Server (version=TLS1_2, > > > cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.1304.16; > > Tue, 8 > > > Aug 2017 19:39:51 + > > > Authentication-Results: spf=pass (sender IP is 66.187.230.42) > > > smtp.mailfrom=ovirt.org; live.com; dkim=none (message not signed) > > > header.d=none;live.com; dmarc=bestguesspass action=none > > > header.from=ovirt.org; > > > Received-SPF: Pass (protection.outlook.com: domain of ovirt.org > &
Re: Fwd: Undeliverable: confirm 286441e8a96ed0284e7691c2ef4d59dee8c9cbfc
K4o9X0UOBIlnepihfU8eTl8lIQXDC41is=;23:H9MT9XjjCxRdHNF9O72fSGhBJyWqtzrgcZYfWBZkghUcrgPmiktc317vS7DAIQ8Y72hXJMaJviCkdTvRNkT9BL/VlYFTHBLxCGF/3HgBnbvmSeDuwrnOo+oOq3MnWeVAjwXx5f+g+5603rzv7jxC8eR7SwrhbkGZ9IBz+ABKWi0= > X-Microsoft-Exchange-Diagnostics: > > 1;SN1NAM04HT150;6:9D6iqC+dpUozipQRvgeNpuutSBmRA45XZe62dCUUr32A1U+SzLu2aDwgxSXML4CBVMlroMqz2E+NzoxlF3O+3Jh+kf+6WZBmRCOfDx83Y+JkY6VmL03RszGP2lx634hnJlwyTv5VMq0yNJ8Ko7tAs5aL40s42SctLTL7R34eM5GbiG2Cj16TwA1RIwZ4DpqgthIeT4G0qvYl1Cr7rljafQgiAicf7U5CXDVc10+to2mzoeFonk9W1Jpa2CBg/4XIoFgfAcMlxtmNYq+XIjh4oh87/bQLUoHqqYeVgHFw2yl9cTO1XHcID/3NFhmsLqXQ+2Hn53fKPEns6ewN/pixCdBwUcrxEDMjMBL83pVjzGgPoIlLKYhD2ApXWbeXWYDxc/Ira4OdRWazcaQEK9seHLQEiVsHM8BZsdiZeIHTuz0TI9JJcPNWMXUSgkoL7lAYuJubTuJGOEBu43kOECkfkmoMrwQjaP8SX5TH7EdKH6/31KCkLSaDQmpJcMJ+00WXM55RAxdjMhZvcXs8uoGWBjAh7OrAmUNH1SB5YgbplLsHmacq9zQEGP+kEU4eENzwbBDgkJhX1+dFzf5lm+HAuqwsDb4YRmC2vXXLK+q84uw89VxI9vBcR0NwMtqPX1t+kiiEqtsrviKHPmLIENOy5RgGDp3+N5Pa0ymNUrMAYahEcXTeJn5jJdy7sz5ZKFuyLrDrYh10uiLcelm6QJrnfKPWBevCiw4GztPsBXMsuO6O+aB5qzBEAl00NVcr9S8BG1m/IO78jYiXTHl5jHxkXyaltf5ro3wQ/1xeDg9iv5czgcNwTuokfP7NLt15VVMBamG9N1adxtbUPVkI1CDgW+XxopOcRr5kP+C1XS481Y4= > X-Microsoft-Exchange-Diagnostics: > > 1;SN1NAM04HT150;5:9/XFvyC4K0r6yO7Vk6O1cb9pQ+b6kkwAx88zNQfaVZhY8V7LCKznaGTirVGtGyJkSeT81ijejwsqaE/omaUmHgrKtIrPEl0kwHl/9lNBzyQKOGKguvbZC4nynIwdwksZgvLPehN0JaVQRKxg9wg0DnNJ9lNYZ461dudOXlMQmO84Pq5F/nTyidrm++ljyNq34CU3Ggz28zH6q68tVPzLIMl+Ku+Wn0LPYonyA4nKdjCF70d8FdMVMUyo91FI8AfRCU/Ip9MyVkadwk/JsAjOY+ULdp6gw3CTi23aoUez99HjdjjYzkwc8JRhAaIktycHWCijkLXXX9BcdhPZIz0a4SxJhBUfk2ylOSUKdaNtFeTPN5biWgN0y7IQlYDbhh4B+95bqHPOkySL5JPH1Z6vF3AC62VCLh+rlAHOWg+1RFPERKfyxfepijGPz/naCm9Ht/qblMhRXIAut4+qMfMj8uiuyhtF584WbM3QfNXunJfqHN2PxNamAcJoIGJojOQ6;24:7lCS+pRbjutn2BgzcX4b9QiG3RBKTnEnfV4dyCB1NYHAJJMHiXzt+asgdiGjY8owRT7dEtdZ8FhtrXkkB+Xn3Sc3CcCsL+38KdeGGzsZ8Ew= > SpamDiagnosticOutput: 1:99 > SpamDiagnosticMetadata: NSPM > X-Microsoft-Exchange-Diagnostics: > > 1;SN1NAM04HT150;7:nJlsV6/t8OOAnKfTJQ03oZqNm/gsNqfM262VoNgEkP+KHbXoPb4IDd4/iwfcKujhasO6/uD3jtQh+mL8UeY6Jo6JmRPqixXGtoLTkFMgocQFkgEksuc64Lwio53SM6zvKfT8DEx+zcv/oXgxee4aBTKVCiRnewkCGXg4zU4Ful1tn//fEkH5TIbu64pMgo9OaOuxDlLD+oMCq0/gzVXjEHgSpFNOVNQJWbESv1Og7pf8p3cNGc4ZO21uAM4eJdzTCXTOCI6EXropDlxIftFod9JT8EphjdPhrVTSsW8xQRTon8DG9cKbfqGV6gZJi0avlS/Jl3xog+AS1frWUIPBmb2jHLTF1ROM2TJxbIgTbhCnjdgt+0UUnVpRvqd4yGFyjc+BST4NjsliesJUw1X3VsrBWljZpR4/m/CJiMR3f8Q8+Rz18S0tVUsf6+LBFrN0XG2F00R/tSydzDi499Yl4r7wCHPFWMVhQjkTD4e6CVxsXF0u2YvM6kqA0YOpgnsklEYGwCeNOsgw5zTZvATB+Vhp8PHOZar1cpU/Z6sRcIvxLRE9JRMJ4H97ieEpj7iQfy4ugsZDCDlDjMzyXM+05OwthtCqMrxA4uiVy020MmfpNPMU7qf2rkHl1UajhD4lHyZh2kgq5dG56000d1AyVFeLI9HpVDGYJ7iR2PyQssxS9m2RHFzhzWIMY1pC1CjjydHj7+VVQTh9KAOW1hUDeaRRD14mSAuTomBbjnuI7GI/Zy09No78WSSuClw75XI7WvsAbThsb2uyWdQ7zlIV4SIqGMdrs9o6xrbRwYFK7kE= > X-OriginatorOrg: outlook.com > X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2017 19:39:51.7756 > (UTC) > X-MS-Exchange-CrossTenant-Id: 84df9e7f-e9f6-40af-b435- > X-MS-Exchange-CrossTenant-FromEntityHeader: Internet > X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1NAM04HT150 > > -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
OS1 is down for the moment
Hi *, so the cloud where several services are hosted is currently down. So that mean, if you receive this email that: - new website will not auto deploy (ovirt, community.redhat.com, rdo) - main website and associated services is down (rdo, theopensourceway.org) The team is working on it, I will send update when it is back (soon). -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Fwd: Re: [oVirt-Infra] : New Gateway
0001 0001 86a0 ...o > > 0x0050: .. > > > > (The final octet of our customer's IP address is masked in the above > > output because some automatic parsers become confused when multiple IP > > addresses are included. The value of that octet is "36".) > > > > -John > > President > > Nuclearfallout, Enterprises, Inc. (NFOservers.com) > > > > (We're sending out so many of these notices, and seeing so many > > auto-responses, that we can't go through this email inbox effectively. If > > you have follow-up questions, please contact us at n...@nfoe.net.) > > > > Hervé Leclerc > > CTO > > Alter Way > > 227 Bureaux de la colline > > 1 rue Royale - Bât. D > > 92210 Saint-Cloud > > France > > *+33 141168336 <%2B33%20141168336>* > > +33 6 83979598 > > > > > > > > `like a halo in reverse` > > > > > > > > On Wed, Feb 19, 2014 at 10:46 AM, Hervé Leclerc <herve.lecl...@alterway.fr > > > wrote: > > > >> Hello, > >> > >> Our Internet gateway is changing. > >> Could you please change your actual gateway (*89.31.150.249*) on your > >> machines (89.31.150.215 and 216) and vms to *89.31.150.253* > >> Thanks > >> > >> Let us know when this modification is done. > >> > >> Cheers > >> > >> Hervé Leclerc > >> CTO > >> Alter Way > >> 1, rue royale > >> 9 ème étage > >> 92210 St Cloud > >> *+33 1 41 16 83 36 <%2B33%201%2041%2016%2083%2036>* > >> +33 6 83979598 > >> > >> > >> > >> > >> <http://www.alterway.fr/signatures/url/1> > >> > >> > >> > >> > > > -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Download.gluster.org 27 April 2016 postmortem
Le mercredi 27 avril 2016 à 14:39 +0300, Eyal Edri a écrit : > Excellent post-mortem! > > Do you think its worth adding mirrors to gluster repos like oVirt is doing? > [1] > > [1] http://ovirt-infra-docs.readthedocs.org/en/latest/General/Mirror.html That could be a solution. But we have the ressources to host a mirror ourself in the DC, it just need a ip address, and a migration of servers (which is taking a awful lot of time to happen :/ ). One issue we would have with a mirror is on the download stats. This and the need to have a mirrorlist, not sure how that's done on dnf/yum side theses days. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Download.gluster.org 27 April 2016 postmortem
Hi, as promised, here is the post-mortem of the incident, if you would like to see more information, or any remarks, please do not hesitate, since that's the first attempt at it we do. I modelled it based on the example of http://shop.oreilly.com/product/0636920041528.do, as that the book I am reading at the moment (Appendix D). We will formalize that later. Download.gluster.org was not serving file Date: 2016-04-27 Participating people: - misc Summary: Download.gluster.org http server was showing error 403 for all url, which did impact ovirt jenkins jobs, and users using the repository, among others. The server is used to distribute gluster rpms. Impact: - ovirt CI jobs got blocked - user couldn't install gluster Root cause: the underlying block device on rackspace was down for a undiagnosed reason, triggering xfs error on the server and thus 403 on the http level. the root cause of the block device error is for still unknown, no error have been seen on the rackspace status page for this DC. A ticket was opened with rackspace to see what was going on (160427-iad-814), a follow up of this post-mortem will be done if the ticket say something more than "shit happens". Resolution: The whole server was rebooted, and upon reboot, the block device came back. Lessons learned: - what went well: - people notified the admin quickly on irc and on gluster-infra - when we were lucky - the server and block device came back immediately - it failed during business hours of EMEA with misc being on irc (just arrived at the office) - what went bad - we do not have proper HA for the service - we do not have automated monitoring for it - the setup is using 2 blocks device of 120G in lvm, thus making it twice as risky to fail Timeline (in UTC) - 05:39 first error message in the log about XFS error - 08:41 misc is pinged on irc - 08:56 misc ack and diagnose the issue - 09:00 the server and service is back to normal - 09:00 first mail about the problem hit gluster-infra Potential improvement to make: - add monitoring on gluster side - use the centos sig repo on ovirt side - add more sysadmin for gluster - add a redundant service for that - a 2nd download server with a shared gluster backend - migrate the storage to a proper setup with 1 single block device, rather than 2. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Supporting Ansible as another tool for ovirt infra mgmt
Le dimanche 03 avril 2016 à 10:42 +0300, Barak Korren a écrit : > On 3 April 2016 at 10:21, Eyal Edri <ee...@redhat.com> wrote: > > I'd like to ask the team what do you think on $subject, in terms of pros & > > cons. > > > > As you all know we have been using puppet to manage our production infra > > (user access, server configuration,etc... ). > > > > Recently we started looking into migrating our mailman instance into new > > hyper-kitty instance to run on the oVirt DC in PHX. > > It seems that there is no true puppet classes available to install/manage > > mailman3 but there is an Ansible playbook used / written by fedora to deploy > > their instance. > > So the question is should we start using/supporting Ansible as another tool > > to manage our infra and leverage existing playbooks out there to reduce work > > on migration of new services? > > I'm not suggesting to migrate all puppet code into Ansible, but to allow > > using Ansible when it make sense. > > > > Here is what I had in mind with regards to pro/cons: > > Pros > > > > Saving time writing puppet classes for services (if Ansible playbook exists) > > Bringing in new infra members which are interested in Ansbile (maybe publish > > the team in the relevant communities) > > > > > > Cons: > > > > Another tool to support/maintain > > No real support to manage in foreman as we do for puppet (for sure not in > > our old version) > > > > > > > > I'd love to hear your thoughts about it. > > > > As I've already stated elsewhere Ansible is interesting for a number > of reasons, but a dual-tool scenario will not be welcome IMO. > > There is also a lage question of the possibility of replacing Puppet > with Ansible. Puppet is a continues configuration-management system > that monitors servers for configuration drift and repairs it > (deploying missing components in the process), to do that it supports > a declarative language and a master/slave-agent architecture. > The common Ansible usage scenario OTOH seems to be AFAIK a > developer/op launching a deployment task from his laptop. For that > Ansible supports a more imperative syntax and an SSH-based agent-less > architecture. > > IMO, for long-running on-premise infrastructure (Not ad-hoc in "the > cloud") which is what oVirt has and what what it targets, the drift > monitoring approach is more suitable. > > Now, I've hared that that Ansible could also be deployed with agents > and a central server (Tower? Foreman? something else?), but I'm not > sure how mature that deployment scenario is right now, nor wither > existing Ansible code fits that scenario. So in my case, I do prefer the configuration drift stuff, but I just use cron to make it converge if it drift with ansible. Also, while I like the whole concept of configuration converging, servers usually do not go out of convergence by themself, so if the sysadmins do not change it, there isn't less pressure to make it converge. So far, what i do is that each push a on specific server trigger a ansible run, restricted to the server that should be impacted. This is either done with ssh (for manageiq.org), or using saltstack bus (for gluster.org, historical reason), or I also do have cron + local run of ansible (for theopensourceway.org). So while most people likely use it to deploy from laptop, that's kinda not how I envision, even in the cloud. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Supporting Ansible as another tool for ovirt infra mgmt
Le lundi 04 avril 2016 à 11:14 +0900, Marc Dequènes (Duck) a écrit : > Quack, > > On 04/03/2016 04:42 PM, Barak Korren wrote: > > > IMO, for long-running on-premise infrastructure (Not ad-hoc in "the > > cloud") which is what oVirt has and what what it targets, the drift > > monitoring approach is more suitable. > > It is possible to run Ansible on an admin machine with a crontab or > triggered by git pushes or Jenkins. It can report changes/drifts in dry > run mode, but the results are not easy to read. > > I think there is something in Tower for parsing the output and do proper > report but I never could try it > (https://www.ansible.com/security-and-compliance). I wonder when it will > be opensourced. Misc do you have any idea on this (and maybe tested some > of these features)? No idea on when this is gonna be open sourced (I keep asking), and didn't test the feature. However, foreman do support reporting from ansible, IIRC. You can get the list of server from foreman since a long time, you can store result in foreman, and I am quite sure that would be trivial to store facts in foreman. There was a few demo during fosdem and cfgmgmt camp, but I didn't listened as closely as I should (due to breakage and sysadmin stuff during the talk) See http://cfgmgmtcamp.eu/schedule/speakers/DanielLobatoGarcia.html so maybe someone can contact him so he can tell the state of the art regarding integration ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [SOLVED] Temporary Outage - ovirt.org Website
Le lundi 07 mars 2016 à 15:50 +0200, Barak Korren a écrit : > > > > While we can't do much for the first or the 2nd (besides blocking), I > > would suggest that the 3rd ip owner decide to reduce the frequency of > > check from 1 every 2 seconds to 1 every 5 minutes. > > > > Sure, that was us, I just removed the whole thing, I guess you have > your own and better monitoring now. Nope, we don't. nagios is fine, but well, every second is excessive :/ -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [SOLVED] Temporary Outage - ovirt.org Website
Le lundi 07 mars 2016 à 14:13 +0100, Mikey Ariel a écrit : > Hi folks, > > It seems that our logs on OpenShift filled up and the ovirt.org website > was unavailable for a few hours this morning. > > The issue was resolved and the site is back online. So to complete what Mikey said: - we have increased the partition from 1G to 3G (this was done automatically with previous openshift version, so insert my usual rant about changes in *aaS), so we would have more room next time. - a RCA (root cause analysis) show that we got 250 000 requests today from 1 single IP at OVH, around 100 000 from bing bot reindexing the website (on 4 or 5 ip ), and 30 000 requests from a ip of bezeqint.net, with a nagios check_http signature. While we can't do much for the first or the 2nd (besides blocking), I would suggest that the 3rd ip owner decide to reduce the frequency of check from 1 every 2 seconds to 1 every 5 minutes. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: SSL certificates
Le dimanche 06 mars 2016 à 09:12 +, Eyal Edri a écrit : > Adding Mikey, > The site was moved to github pages, so its probably related to the > migration. Nope, taht's not githuab pages, taht's hosted on openshift and use middleman to be generated. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
non html mail version of jira notification
Hi, Am I the only who disable html mail (for security reason), and who would like to see a non html mail notifcation from Jira ? Could we open a RFE to the provider (since we can't patch it, not opensource...) ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: creating a new ovirt-italian mailing list
Le vendredi 15 janvier 2016 à 11:39 -0500, Einav Cohen a écrit : > Hi, > > There seems to be a need for an Italian-focused oVirt mailing > list; from what I know, there are ~23 members in the ovirt > community that would benefit from it, mainly in order to discuss > Italian localization issues in oVirt. > There is a similar mailing list for Fedora, for example [1]. > > Can you please create a new "ovirt-italian" / "italian" / > "italian-users" / "it-users" (@Mikey/Francesco - feel free > to chime in with your preferred name) mailing list? so far, we have users-pt, and users, so that would be users-it. http://lists.ovirt.org/mailman/listinfo Also, I need to have a list of people to administrate it :) -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: spam on mailling list
Le mercredi 07 octobre 2015 à 14:43 +0300, Eyal Edri a écrit : > thanks michael, > do we have any documentation on our spam filters/email conf somewhere on > the wiki/servers? nope. this is not in puppet or anything, as i am mostly only able to do firefighting on ovirt side for now, as i focus on other less staffed product for the time being :/ > e. > > On Mon, Sep 21, 2015 at 12:58 PM, Michael Scherer <msche...@redhat.com> > wrote: > > > Hi, > > > > it seems we are targetted by the same person as on > > > > http://smoogespace.blogspot.fr/2015/09/note-to-mailing-list-admins-large-scale.html > > > > so I am gonna deploy the same module as Fedora on list.ovirt.org > > -- > > Michael Scherer > > Sysadmin, Community Infrastructure and Platform, OSAS > > > > > > ___ > > Infra mailing list > > Infra@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/infra > > > > > > -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Fwd: wiki protected pages
Le jeudi 24 septembre 2015 à 15:00 +0300, Max Kovgan a écrit : > hi. > I want to be able to edit the protected pages, and so is Nadav ( > ngol...@redhat.com). > > who do we contact? I would say brian likely know. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
spam on mailling list
Hi, it seems we are targetted by the same person as on http://smoogespace.blogspot.fr/2015/09/note-to-mailing-list-admins-large-scale.html so I am gonna deploy the same module as Fedora on list.ovirt.org -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Failure to request wiki account
Le mercredi 09 septembre 2015 à 11:16 -0400, Brian Proffitt a écrit : > Michael Scherer is the person who handles this, and I have already > forwarded the problem to him. So the ip of the wiki changed again. A proper fix is to deploy proper username/password security on smtp, along a certificate, but since the wiki is supposed to be out soon, I never bothered. Instead, i did add a fast script to add the ip if it change. It is in /usr/local/sbin/fix_postfix.py on lists.ovirt.org, I plan to make it run by cron every hour, that should be good enough for the short term. (also, top post is bad) -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Mailman status
Le mardi 08 septembre 2015 à 17:31 +0300, Eyal Edri a écrit : > Btw, > Anyone from the team will be able to help in migrating mailman to a new > server on phx lab? > maybe to mailman3? I would prefer to have a bit more experience with mailman3 before switching. But I didn't had time yet to play with it, nor even read any documentation yet, and quite busy on others migra One idea is do like Fedora, just move 1 single ML of lower importance to see how it goes, and then move one after the others. This can be done with some postfix tricks. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Mailman status
Le jeudi 03 septembre 2015 à 17:08 -0700, Karsten Wade a écrit : > All: > > Not sure who else is getting this, but since last week I've been > getting a subscription request to workshop-nov11-owner about every ten > minutes. > > Can someone go in an disable that list for subscription attempts? Same > with the other workshop list, and it can also be hidden from the > listinfo page. > > I personally don't like completely vaporizing a historical record, but > I think hide and disable will work. So, I commented the alias in /etc/aliases, ran newalias. Not sure if we should hide from the listinfo page, because then, we are vaporizing a list record from google, since there is no link anymore :) We should also take a look at al list and see which we want to keep and which we do not want anymore. -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [Fwd: Re: yahoo cancellation blocker]
Le mardi 01 septembre 2015 à 11:19 +0200, David Jaša a écrit : > Forwarded Message > From: Michael Scherer <msche...@redhat.com> > To: David Jaša <dj...@redhat.com> > Cc: devel-ow...@ovirt.org > Subject: Re: yahoo cancellation blocker > Date: Tue, 01 Sep 2015 10:35:14 +0200 > > Le mardi 01 septembre 2015 à 09:58 +0200, David Jaša a écrit : > > Hi, > > > > would you mind setting up a filter blocking fake event cancellations > > from Yahoo servers, as Michal diagnosed it last time? > > Hi, > Can you write this to infra@ovirt.org ? So now, this is public, i was hoping that someone would know what this is about, but since there is a lack of answer, I guess I need to ask ask for clarification. David, can you explain a bit more ? I do not know what is it about, nor even whose Michal we are speaking about ( I assume Michal Skrivanek, so correct me if i am wrong ). Do you have a reference on this issue, or something ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: access to linode01.ovirt.org
Le jeudi 03 septembre 2015 à 11:38 +0300, Tolik Litovsky a écrit : > yes but this mean that I wait 30 minute every time . > the problem is not the frequence , its the delay . How long does it take to run the job, as it could be run more than every 30 minutes -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: access to linode01.ovirt.org
Le jeudi 03 septembre 2015 à 14:11 +0200, David Caro a écrit : > On 09/03, Michael Scherer wrote: > > Le jeudi 03 septembre 2015 à 11:38 +0300, Tolik Litovsky a écrit : > > > yes but this mean that I wait 30 minute every time . > > > the problem is not the frequence , its the delay . > > > > How long does it take to run the job, as it could be run more than every > > 30 minutes > > The job takes ~ 10 minutes, sometimes a bit more than 20 Is there a way to stop it if nothing have to be done ? ( ie, run it every 5 minutes, but continue only if something have to be changed ? ) if not, maybe using incron could be a solution, but i do not know how reliable and scalable it would be :/ -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Greylisting
Hi, in a attempt to curb spam and one that would be compatible with the low ressources of the list server, i setup a simple greylisting setup. I am currently watching to whitelist what should be whitelisted ( so far, gerrit.ovirt.org ), but if your mail take too much time to appear, please ping me on irc with enough info so I can look. I also found out that bugzilla still send email to bugs@ovirt , and this is not working anymore, someone has a idea ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: migration of services from linode server to phx2 lab
Le mardi 16 juin 2015 à 03:38 -0400, Eyal Edri a écrit : - Original Message - From: Karsten Wade kw...@redhat.com To: infra@ovirt.org Sent: Tuesday, June 16, 2015 6:29:04 AM Subject: Re: migration of services from linode server to phx2 lab -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi, Between myself (original linode01.ovirt.org admin) and Michael (misc, aka knower-of-stuff), what can we do to get us off this Linode instance? From what I can tell, at least Mailman is running from there. If we still need service failover, can we switch to another Red Hat-provided service such as an OpenShift or OpenStack instance? Hi Karsten, I know this has been taking for too long, but unfortunately there are many reasons and obstacles that prevented us from moving, which i'll explain below, while there is a risk right now of moving services to PHX, but i think we can try. Blocking items: - were missing more hypervisors to the production DC ( we finally got them installed last week, and are now in final stages of bringing them up) - storage issue - currently NFS storage is quite slow, we are testing moving to a mixed mode of local + nfs for the jenkins slaves, though might be that the production service won't get affected too much - worth a try. - lack of monitoring for servers, which might add risk if we hit an issue. there are some other issues, but not blocking imo. Here is what needs to be done (per service to migrate off linode), and we'd appreciate any help given. 1. create new VM on production DC in PHX (assign dns/ip/etc.. ) 2. create puppet manifests to manage that vm, so it will be easy to reproduce it and maintain it 3. install the relevant software on it (for e.g mailman/ircbot/etc...) 4. test to see it works 5. do the actual migration (downtime of existing service, and bringing up the new one) So, last time I did look, the puppet setup was a bit unfinished rather overkill ( using r10k to deploy count as overkill when you do not have stage/prod/qa, at least for me ). Did stuff changed, or should first some effort be made to fix/automate/improve the puppet setup ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: spam email requests for infra list - suggestions
Le lundi 15 juin 2015 à 03:52 -0400, Eyal Edri a écrit : - Original Message - From: Itamar Heim ih...@redhat.com To: Michael Scherer msche...@redhat.com, infra infra@ovirt.org Sent: Friday, June 12, 2015 1:41:03 PM Subject: Re: spam email requests for infra list - suggestions On 06/12/2015 01:12 PM, Michael Scherer wrote: Le jeudi 11 juin 2015 à 23:58 +0300, Itamar Heim a écrit : On 06/10/2015 11:20 AM, Eyal Edri wrote: Hey, since i got added as admin to the infra list, i started seeing multiple spam requests to get accepted to the list. something happened past week or so, that spam emails started being sent or not filtered to -owner's of mailing lists. I don't remember seeing that before. Or they were filtered before on your mailbox and sent since the beginning ? Looking at the settings, there is: forward_auto_discards = 1 (not sure where to find it in the UI). Could setting this to false/0 be a solution ? its under privacy/sender filters Should messages from non-members, which are automatically discarded, be forwarded to the list moderator? automatically discarded here means known spammers with fixed sender addresses, not spam. i've set that, but i'm still getting spam requests. the problem isn't with messages sent to the list, the problem is spammers sending requests to be added to the list, and i didn't see any option to block, other than maintaining manually a blacklist of emails. Are the emails tagged by SA in some way ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: spam email requests for infra list - suggestions
Le jeudi 11 juin 2015 à 23:58 +0300, Itamar Heim a écrit : On 06/10/2015 11:20 AM, Eyal Edri wrote: Hey, since i got added as admin to the infra list, i started seeing multiple spam requests to get accepted to the list. something happened past week or so, that spam emails started being sent or not filtered to -owner's of mailing lists. I don't remember seeing that before. Or they were filtered before on your mailbox and sent since the beginning ? Looking at the settings, there is: forward_auto_discards = 1 (not sure where to find it in the UI). Could setting this to false/0 be a solution ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Jira not public ?
Le mardi 02 juin 2015 à 06:12 -0400, Eyal Edri a écrit : what do you mean not public? it is avaliable on https://ovirt-jira.atlassian.net I need to login to access this page and so does clicking on https://ovirt-jira.atlassian.net/browse/OVIRT-289 ( also, sorry for the double mail, forgot to put the list in CC ) we havn't yet solved the issue of opening tickets easily and we're working on it. the idea is to add new support-in...@ovirt.org email to allow people to open tickets automatically. also, since JIRA doesn't support open-id, the users currently are managed internally, so that's also an issue we need to resolve. did you meant something else? e. - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Tuesday, June 2, 2015 1:08:56 PM Subject: Jira not public ? Hi, I just noticed that our Jira is not public. This seems quite a rather bad idea for a open source project, so can someone fix that ? -- Michael Scherer Open Source and Standards, Sysadmin ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Jira not public ?
Hi, I just noticed that our Jira is not public. This seems quite a rather bad idea for a open source project, so can someone fix that ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Jira not public ?
Le mardi 02 juin 2015 à 07:06 -0400, Eyal Edri a écrit : try now.. https://ovirt-jira.atlassian.net/secure/RapidBoard.jspa?rapidView=3projectKey=OVIRT yep, that work. thanks :) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Logwatch for linode01.ovirt.org (Linux)
Le mercredi 27 mai 2015 à 11:41 +0200, David Caro a écrit : Misc? it was just the one proposed as update by yum update -y I guess it doesn't matter much since linode boot on a different one ? On 05/27, Sandro Bonazzola wrote: Il 27/05/2015 09:22, logwa...@linode01.ovirt.org ha scritto: ### Logwatch 7.3.6 (05/19/07) Processing Initiated: Wed May 27 03:22:56 2015 Date Range Processed: yesterday ( 2015-May-26 ) Period is day. Detail Level of Output: 0 Type of Output: unformatted Logfiles for Host: linode01.ovirt.org ## - httpd Begin Requests with error response codes 400 Bad Request /: 1 Time(s) 404 Not Found //administrator/components/com_jinc/classe ... pload_image.php: 1 Time(s) //administrator/components/com_jnewsletter ... pload_image.php: 2 Time(s) //dompdf/dompdf.php: 2 Time(s) //dompdf/dompdf.php?input_file=http://www. ... /.mods//bt.php?: 1 Time(s) //dompdf/dompdf.php?input_file=http://www. ... /.mods//sh.txt?: 1 Time(s) //force-download.php?file=../wp-config.php: 1 Time(s) //index.php?option=com_jdownloadsItemid=0view=upload: 2 Time(s) //wp-admin/admin-ajax.php?action=revolutio ... ./wp-config.php: 1 Time(s) //wp-admin/admin-ajax.php?action=revslider ... ./wp-config.php: 1 Time(s) //wp-content/force-download.php?file=../wp-config.php: 1 Time(s) //wp-content/plugins/wp-slimstat-ex/lib/of ... =cpx.php.phpgif: 3 Time(s) //wp-content/plugins/wp-slimstat-ex/lib/of ... hp?name=cpx.php: 3 Time(s) //wp-content/plugins/wp-slimstat-ex/lib/of ... p?name=petx.php: 3 Time(s) //wp-content/plugins/wp-slimstat-ex/lib/of ... petx.php.phpgif: 3 Time(s) /ACG-NWL/assetmanager/assetmanager.asp: 1 Time(s) /Diagnostics.asp: 119 Time(s) /Releases: 2 Time(s) /Ringing.at.your.dorbell!: 119 Time(s) /_h5ai/client/images/app-16x16.ico: 30 Time(s) /admin.php: 8 Time(s) /admin/: 7 Time(s) /admin/MembersAreaManager/components/Edito ... ssetmanager.asp: 1 Time(s) /admin/board: 1 Time(s) /admin/fckeditor/editor/filemanager/connec ... rrentFolder=%2F: 3 Time(s) /admin/index.php?route=common/login: 1 Time(s) /admin/login.php: 7 Time(s) /administrator/index.php: 32 Time(s) /apple-touch-icon-precomposed.png: 3 Time(s) /apple-touch-icon.png: 3 Time(s) /applications/ClassifiedListingsManager/co ... ssetmanager.asp: 1 Time(s) /assetmanagerOLD/assetmanager.asp: 1 Time(s) /bitrix/admin/index.php?lang=en: 7 Time(s) /blog/wp-admin/: 15 Time(s) /board: 2 Time(s) /browserconfig.xml: 1 Time(s) /category/news/feed: 1 Time(s) /category/news/feed/: 18 Time(s) /dbadmin/: 1 Time(s) /editor/editor/filemanager/connectors/asp/ ... rrentFolder=%2F: 1 Time(s) /editor/editor/filemanager/connectors/aspx ... rrentFolder=%2F: 1 Time(s) /editor/editor/filemanager/connectors/php/ ... rrentFolder=%2F: 1 Time(s) /editor/filemanager/connectors/asp/connect ... rrentFolder=%2F: 1 Time(s) /editor/filemanager/connectors/aspx/connec ... rrentFolder=%2F: 1 Time(s) /editor/filemanager/connectors/php/connect ... rrentFolder=%2F: 1 Time(s) /favicon.ico: 747 Time(s) /fckeditor/editor/filemanager/connectors/a ... rrentFolder=%2F: 1 Time(s) /fckeditor/editor/filemanager/connectors/p ... rrentFolder=%2F: 1 Time(s) /index.php/component/user/?task=register: 3 Time(s) /index.php/component/users/?view=registration: 3 Time(s) /index.php?gf_page=upload: 1 Time(s) /index.php?option=com_userview=register: 3 Time(s) /listinfo/board: 1 Time(s) /m/pipermail/node-devel/2012-October/000348.html: 2 Time(s) /m/pipermail/users/2013-August/015569.html: 1 Time(s) /m/pipermail/users/2013-November/017756.html: 1 Time(s) /mailman/listinfo=: 1 Time(s) /mobile/pipermail/node-devel/2012-October/000348.html: 2 Time(s) /mobile/pipermail/users/2014-April/023081.html: 1 Time(s) /myadmin/: 1 Time(s) /old/wp-admin/: 15 Time(s) /ovirt/editor/filemanager/connectors/asp/c ... rrentFolder=%2F: 1 Time(s) /ovirt/editor/filemanager/connectors/aspx/ ... rrentFolder=%2F: 1 Time(s) /ovirt/fckeditor/editor/filemanager/connec ... rrentFolder=%2F: 3 Time(s)
Re: Bounce on users lists
Le vendredi 16 janvier 2015 à 08:23 +0100, Sandro Bonazzola a écrit : Il 15/01/2015 18:31, Michael Scherer ha scritto: Hi, Itamar, by the proxy of Brian, did asked me to look on the bounce issue we have on the users lists. So after a few hours of careful log reading, here is my finding. The bounce situation - We (ml admin) get on a regular basis people who get unsubscribed and message about bounce. People being unsubscribed automatically is bad(tm), and bounces are annoying. Investigation - A first look show that our mails are bounced as they are marked as spam by Google. Google doc on the matter do not give much, some people point to using dkim, spf, etc. But spf is not for us, but for the sender, and dkim is not ml friendly, afaik, and requires upstream support if I understood well. Not all mails are bounced, which is good. That mean the ip is not problematic. So I took a few hours to look on every bounce and roughly, there is 2 groups. Group 1 First group is that all mail from the same poster on the users list have bounced at Google. Out of the 16 mail he sent, 16 have been rejected by Google. I have no idea why, I suspect the spf policy, but it did looked ok. None of the mail of answer had a issue, so that's likely not a content problem. However, the ip address of the sender is in the SORBS blacklist, so that's likely what trigger Google spam filter. Not much we can do, besides contacting him, which I will do. Group 2 Roughly, that's mail in this thread : http://lists.ovirt.org/pipermail/users/2015-January/030494.html and the mails from Sandro : http://lists.ovirt.org/pipermail/users/2015-January/030420.html http://lists.ovirt.org/pipermail/users/2015-January/030423.html Common point, use of goo.gl and ur1.ca. It turn out that both domain are flagged as URI spam, since that's used by spammer to hide their link. So I suspect that Gmail started to learn about them as spam, as the rest of the world did : http://multirbl.valli.org/lookup/ur1.ca.html http://multirbl.valli.org/lookup/goo.gl.html Again, not much we can do, besides asking to people to not use these services ( which is not gonna work I think ). I may try to use bit.ly red.ht instead of goo.gl. Can we provide our own url shortener on ovirt.org? That should avoid blacklisting. I do not think bit.ly is gonna change much. It is likely abused for the same reason by the same people. And the url are too complicated to be sent sometime, so we cannot just avoid them at all. I also pondered about adding a url shortener on ovirt.org. Besides the load on admin team it create, I think it would have the same issue as the others after some time, and so we would need to add some authentication, which start to make thing a bit complicated. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Bounce on users lists
Le vendredi 16 janvier 2015 à 09:31 +0100, Sandro Bonazzola a écrit : Il 16/01/2015 09:18, Michael Scherer ha scritto: Le vendredi 16 janvier 2015 à 08:23 +0100, Sandro Bonazzola a écrit : Il 15/01/2015 18:31, Michael Scherer ha scritto: Hi, Itamar, by the proxy of Brian, did asked me to look on the bounce issue we have on the users lists. So after a few hours of careful log reading, here is my finding. The bounce situation - We (ml admin) get on a regular basis people who get unsubscribed and message about bounce. People being unsubscribed automatically is bad(tm), and bounces are annoying. Investigation - A first look show that our mails are bounced as they are marked as spam by Google. Google doc on the matter do not give much, some people point to using dkim, spf, etc. But spf is not for us, but for the sender, and dkim is not ml friendly, afaik, and requires upstream support if I understood well. Not all mails are bounced, which is good. That mean the ip is not problematic. So I took a few hours to look on every bounce and roughly, there is 2 groups. Group 1 First group is that all mail from the same poster on the users list have bounced at Google. Out of the 16 mail he sent, 16 have been rejected by Google. I have no idea why, I suspect the spf policy, but it did looked ok. None of the mail of answer had a issue, so that's likely not a content problem. However, the ip address of the sender is in the SORBS blacklist, so that's likely what trigger Google spam filter. Not much we can do, besides contacting him, which I will do. Group 2 Roughly, that's mail in this thread : http://lists.ovirt.org/pipermail/users/2015-January/030494.html and the mails from Sandro : http://lists.ovirt.org/pipermail/users/2015-January/030420.html http://lists.ovirt.org/pipermail/users/2015-January/030423.html Common point, use of goo.gl and ur1.ca. It turn out that both domain are flagged as URI spam, since that's used by spammer to hide their link. So I suspect that Gmail started to learn about them as spam, as the rest of the world did : http://multirbl.valli.org/lookup/ur1.ca.html http://multirbl.valli.org/lookup/goo.gl.html Again, not much we can do, besides asking to people to not use these services ( which is not gonna work I think ). I may try to use bit.ly red.ht instead of goo.gl. Can we provide our own url shortener on ovirt.org? That should avoid blacklisting. I do not think bit.ly is gonna change much. It is likely abused for the same reason by the same people. And the url are too complicated to be sent sometime, so we cannot just avoid them at all. I also pondered about adding a url shortener on ovirt.org. Besides the load on admin team it create, I think it would have the same issue as the others after some time, and so we would need to add some authentication, which start to make thing a bit complicated. Not sure if authentication will work as expected, but this one seems quite simple to configure and deploy: https://github.com/mrtazz/katana Not that keen on adding it on the infra, but could work for openshift maybe ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: wiki: DISPLAYTITLE macro stopped working
Le vendredi 16 janvier 2015 à 10:39 +0100, Sandro Bonazzola a écrit :
Looks like the DISPLAYTITLE macro stopped working here:
http://www.ovirt.org/OVirt_3.5.1_Release_Notes
{{DISPLAYTITLE:oVirt 3.5.1 Release Notes}}
Any hint on what's going on?
it is fine for me
--
Michael Scherer
Open Source and Standards, Sysadmin
signature.asc
Description: This is a digitally signed message part
___
Infra mailing list
Infra@ovirt.org
http://lists.ovirt.org/mailman/listinfo/infra
Bounce on users lists
Hi, Itamar, by the proxy of Brian, did asked me to look on the bounce issue we have on the users lists. So after a few hours of careful log reading, here is my finding. The bounce situation - We (ml admin) get on a regular basis people who get unsubscribed and message about bounce. People being unsubscribed automatically is bad(tm), and bounces are annoying. Investigation - A first look show that our mails are bounced as they are marked as spam by Google. Google doc on the matter do not give much, some people point to using dkim, spf, etc. But spf is not for us, but for the sender, and dkim is not ml friendly, afaik, and requires upstream support if I understood well. Not all mails are bounced, which is good. That mean the ip is not problematic. So I took a few hours to look on every bounce and roughly, there is 2 groups. Group 1 First group is that all mail from the same poster on the users list have bounced at Google. Out of the 16 mail he sent, 16 have been rejected by Google. I have no idea why, I suspect the spf policy, but it did looked ok. None of the mail of answer had a issue, so that's likely not a content problem. However, the ip address of the sender is in the SORBS blacklist, so that's likely what trigger Google spam filter. Not much we can do, besides contacting him, which I will do. Group 2 Roughly, that's mail in this thread : http://lists.ovirt.org/pipermail/users/2015-January/030494.html and the mails from Sandro : http://lists.ovirt.org/pipermail/users/2015-January/030420.html http://lists.ovirt.org/pipermail/users/2015-January/030423.html Common point, use of goo.gl and ur1.ca. It turn out that both domain are flagged as URI spam, since that's used by spammer to hide their link. So I suspect that Gmail started to learn about them as spam, as the rest of the world did : http://multirbl.valli.org/lookup/ur1.ca.html http://multirbl.valli.org/lookup/goo.gl.html Again, not much we can do, besides asking to people to not use these services ( which is not gonna work I think ). Conclusion --- If the core issue is people are kicked out due to bounce, we can look at raising the threshold on mailman ( as proposed by Brian ), while at the same time trying to reduce the number of bounce ( ie, a root cause investigation on each bounce when we see issue ). First part is easy ( I think ), second is not hard but we need to have someone to look at log on a regular basis so that's taking some time. As a side note, our spamassasin setup was blacklisted from the DNS BL we used ( due to our use of the dns of linode.com : http://uribl.com/refused.shtml ), thus reducing his efficiency. I did fixed that by setting a local cache, following the page I gave. If anything weird happen, please tell us :) Anyone has a opinion or a idea ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le mercredi 10 décembre 2014 à 10:41 +0100, Michael Scherer a écrit : Le mardi 09 décembre 2014 à 18:00 -0500, Brian Proffitt a écrit : - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Monday, December 8, 2014 8:07:33 AM Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Monday, December 8, 2014 1:03:01 PM Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : fyi, ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. REASON: === after logging into the openshift gear, seems like there was 99.9% space taken. RESOLUTION: === deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using 'ctl_app restart'. space is still not great - at 84%, so we'll need to think soon how to resolve this. wiki is up now and seems to working well. infra - we should look into migrating to a new gear maybe? or extend again the existing one. Happened again today. I still maintain that we should move to a dedicated hosting on the phx2 cluster :) ( or request quota extensions ). whether if we'll migrate or not, that will take time and won't happen until the ILO issue will be resolved. for now we need to extend our volume there ASAP, cause it will happen again soon, not too many log files are there to delete/gzip. Agreed. We did request already a quota extension, IIRC, so I guess that was not sufficient ? Filled the form to get more, will tell you when I have a response. -- Michael Scherer Open Source and Standards, Sysadmin ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra Went out again this afternoon (12/9), but managed to squeeze out a little more disk space running rhc app tidy. (Thanks to Jason and Justin for the assist!) If there is any way I can help expedite more disk quota, please let me know! It take a few days. Last time we did ask, it was request = go to marketing for approval = go to the sysadmins team for action. As long as this in the marketing side, I have no visibility ( but can ask ). On the openshift sysadmin side, I know where to ping :) So, today it was full again, and openshift people did contact me. We increased the quota, and while on it, I did the dump/restore needed to clean the ever increasing ibdata1 file ( http://www.percona.com/blog/2013/08/20/why-is-the-ibdata1-file-continuously-growing-in-mysql/ ). It took a few hours to process, and since things would not be fun without it, my internet connexion started to behave badly. But now, we are down to 6G out of 10G, so wait and see. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le mardi 09 décembre 2014 à 18:00 -0500, Brian Proffitt a écrit : - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Monday, December 8, 2014 8:07:33 AM Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Monday, December 8, 2014 1:03:01 PM Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : fyi, ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. REASON: === after logging into the openshift gear, seems like there was 99.9% space taken. RESOLUTION: === deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using 'ctl_app restart'. space is still not great - at 84%, so we'll need to think soon how to resolve this. wiki is up now and seems to working well. infra - we should look into migrating to a new gear maybe? or extend again the existing one. Happened again today. I still maintain that we should move to a dedicated hosting on the phx2 cluster :) ( or request quota extensions ). whether if we'll migrate or not, that will take time and won't happen until the ILO issue will be resolved. for now we need to extend our volume there ASAP, cause it will happen again soon, not too many log files are there to delete/gzip. Agreed. We did request already a quota extension, IIRC, so I guess that was not sufficient ? Filled the form to get more, will tell you when I have a response. -- Michael Scherer Open Source and Standards, Sysadmin ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra Went out again this afternoon (12/9), but managed to squeeze out a little more disk space running rhc app tidy. (Thanks to Jason and Justin for the assist!) If there is any way I can help expedite more disk quota, please let me know! It take a few days. Last time we did ask, it was request = go to marketing for approval = go to the sysadmins team for action. As long as this in the marketing side, I have no visibility ( but can ask ). On the openshift sysadmin side, I know where to ping :) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : fyi, ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. REASON: === after logging into the openshift gear, seems like there was 99.9% space taken. RESOLUTION: === deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using 'ctl_app restart'. space is still not great - at 84%, so we'll need to think soon how to resolve this. wiki is up now and seems to working well. infra - we should look into migrating to a new gear maybe? or extend again the existing one. Happened again today. I still maintain that we should move to a dedicated hosting on the phx2 cluster :) ( or request quota extensions ). -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Monday, December 8, 2014 1:03:01 PM Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : fyi, ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. REASON: === after logging into the openshift gear, seems like there was 99.9% space taken. RESOLUTION: === deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using 'ctl_app restart'. space is still not great - at 84%, so we'll need to think soon how to resolve this. wiki is up now and seems to working well. infra - we should look into migrating to a new gear maybe? or extend again the existing one. Happened again today. I still maintain that we should move to a dedicated hosting on the phx2 cluster :) ( or request quota extensions ). whether if we'll migrate or not, that will take time and won't happen until the ILO issue will be resolved. for now we need to extend our volume there ASAP, cause it will happen again soon, not too many log files are there to delete/gzip. Agreed. We did request already a quota extension, IIRC, so I guess that was not sufficient ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [OUTAGE] ovirt.org wiki outage report [07/12/14]
Le lundi 08 décembre 2014 à 13:28 +0100, Michael Scherer a écrit : Le lundi 08 décembre 2014 à 06:23 -0500, Eyal Edri a écrit : - Original Message - From: Michael Scherer msche...@redhat.com To: infra@ovirt.org Sent: Monday, December 8, 2014 1:03:01 PM Subject: Re: [OUTAGE] ovirt.org wiki outage report [07/12/14] Le dimanche 07 décembre 2014 à 03:25 -0500, Eyal Edri a écrit : fyi, ovirt.org wiki page was down today from 01:33:07 IST till 10:22 IST. REASON: === after logging into the openshift gear, seems like there was 99.9% space taken. RESOLUTION: === deleted some logs from $OPENSHIFT_MYSQL_DB_LOG_DIR, and restarted mysql using 'ctl_app restart'. space is still not great - at 84%, so we'll need to think soon how to resolve this. wiki is up now and seems to working well. infra - we should look into migrating to a new gear maybe? or extend again the existing one. Happened again today. I still maintain that we should move to a dedicated hosting on the phx2 cluster :) ( or request quota extensions ). whether if we'll migrate or not, that will take time and won't happen until the ILO issue will be resolved. for now we need to extend our volume there ASAP, cause it will happen again soon, not too many log files are there to delete/gzip. Agreed. We did request already a quota extension, IIRC, so I guess that was not sufficient ? Filled the form to get more, will tell you when I have a response. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: oVirt Wiki won't reset a password
Le mardi 04 novembre 2014 à 16:26 +0100, David Caro Estevez a écrit : On 11/04, Ondřej Svoboda wrote: Hello, I have been trying to recover my password to the Wiki so I could update the outdated IPv6 feature page. It seems that our MediaWiki is not able to send any mail (and I cannot reproduce the error for another 24 hours). Could someone help me out? David? :-) I'm at the openstack summit and I don't have too much time, so far I've been able to see that the email is not being sent because a realy error. I'll look when I have some time, but maybe someone else (misc?) can check sooner I already fixed. The problem is that we are using ip based whitelist, and each time the gears change ip, we are screwed. I did add the new ip, but I am also at the summit so I will defer a proper solution to later. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Moving the wiki
Le lundi 20 octobre 2014 à 11:05 -0400, Barak Korren a écrit : There has been some talk recently about moving the ovirt.org wiki to a dedicated VM in the PHX data-center. I would like to open this up to some discussion. Here is the current situation as far as I could gather, more info and comment are welcome. What do we have now --- MediaWiki (Which version? eedri told me its a rather old one) 1.18 ( or 1.19 ) PHP (Which version?) 5.3.3 MySQL 5.1 All running in a single (?) 'large' OpenShift gear. yes Our OpenShift account is classified as 'silver' (is it?) thereby granting us gears with 6GB storage instead of 1GB) yes. I think we even already have 10G Why do we want to migrate? -- We occasionally have a problem where the site goes down. This seems to be caused by one of: 1. The OpenShift gear runs out of space 2. The MySQL DB gets stuck with no errors in the logs (Does restarting it resolve it?) it usually was out of memory issue. besides the problem, one reason to go to a more traditional setup for me is that, being traditional, we have more freedom. For example, installing varnish directly, having access to log without a middleman, ease of backup. And the capacity to use vanilla mediawiki. Why not to migrate? --- 1. Migrating the wiki to PHX VM will make the infra team have to manage the wiki hosting infrastructure. While one may claim that this is not complicated and that this work needs to also be done when the wiki is hosted on OpenShift, there are still many things that the OpenShift maintainers do for us such as: - Keeping the webservers updated there is just yum upgrade -y to do. I do not see that much as a hindrance. - Managing selinux There is nothing to manage, selinux work out of the box on RHEL. Also, due to the nature of openshift, the policy will only protect the host, but you would still be able to access network and this kind access. While with our own setup, we can have a tailored policy or firewall. - Enablign automatic scale-up We have no scale up for that, and due to mediawiki itself, we either have to patch it to not use the filesystem ( people suggested to use s3 ), or wait until fs is shared on openshift. 2. There are security concerns with having a public-facing (outdated?) PHP application running on a VM in the same network where our build and CI servers run. (I might be too paranoid here, having had one of my own sites defaced recently, but OpenShift makes it easy to create a new gear and git push the code to get back up and running, with our own VM, forensics and cleanup might be more complicated) I do not see how openshift will be easier. We can make a snapshot of the VM in ovirt too, afaik, so the forensic wouldn't be harder ( and in fact, would preserve the memory, which is rather important ). Known infra issues with existing configuration -- 1. The MySQL DB was setup without 'innodb_file_per_table' turned on, this can impact DB performance. To resolve this, one need to dump and import the entire DB. Things we can try (Besides migrating) - 1. Place ovirt.org behind a caching reverse-proxy CDN like cloudflare, that can mask some of our downtime. 2. Dump and import the DB to rebuild and optimize the DB files we need to have a bit more space to operate, that's the issue. 3. Rebuild the wiki in a new gear to get rid of possibly accumulating cruft 4. Upgrade the MySQL to 5.5 (Or whatever latest supported by OpenShift) We can't do that easily. 5. Upgrade MediaWiki We would have to rediff some of the patchs, I would rather start from scratch. 6. Add a redundant MySQL/Wiki server using MySQL replication Not working due to gears isolation. IE, 2 gears cannot communicate that easily. This would be doable with a special embedded cartridge. 7. Trim the wiki history (AFAIK MediaWiki saves every edit ever, but one can maybe use export/import to get rid of some) Why drop history, that's like dropping git history :/ 8. Try to come up with a way to spread the Wiki across multiple OpenShift gears Rather hard to do, especially since we are not root. 9. Tune DB parameters (is it possible to do on OpenShift?) No, since the mysql is managed by openshift we are not root. I eagerly await your comments, I still think the easiest way is to host our own setup. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Cannot loging to ovirt wiki
Le jeudi 18 septembre 2014 à 10:42 +0200, David Caro a écrit : On Thursday, September 18, 2014 01:10:15 AM Michael Scherer wrote: Le mercredi 17 septembre 2014 à 15:37 -0400, R P Herrold a écrit : On Wed, 17 Sep 2014, Michael Scherer wrote: As I said in the past, the plan wouldn't work. To have 2 gears communicate, we need to have them setup in a specific way, not just 2 gears in the same account. If one is moved to another node, we need to have a specific triggers on the webserver gear to trigger a potential configuration change. Why not just point the two through a pair of keyed access openvpn links, each to a fixed (and routing) central hub? MySQL will communicate just fine across a network fabric hub 10.0.0.1 10.0.1.1 /\ / \ 10.0.0.2 10.0.1.2 gear A gear B (the wiki) (the MySQL server) The hub just routes 10.0.1 and 10.0.0 back and forth Nothing changes, save re-establishment of an openvpn link when a 'spoke' moves I would slightly be against the idea because : 1) we do not root access in the gears 2) the firewall will likely not be open for that from the gear to external world 3) one of the main selling point of using openshift online was that we do not have to manage the platform aspect. Adding openvpn to bypass the platform is kinda managing a different platform than what we have, and kinda negate the main advantage of using openshift. 4) we would have to manage the hub ( so need to manage 1 more server ), so we could as well manage mysql and the wiki on the server and that's it ? If we must stretch the platform to its limit to make it do what we want, I think we should accept that what we want is not what we have. Again, i think openshift is a fine product when you use it with software made for the platform ( ie, aware of the scaling requirement, aware of the variable for integration, stateless if possible ). But currently, it is: - not integrated with puppet ( so we have 2 identity store ) - not integrated with icinga ( so it has its own monitoring ) - no backups made by ovirt infra ( but made by openshift ops ) - various space issue ( with a quite complex solution ) We can surely solve each of this with enough hack. I can surely run puppet inside the gear if I want, running a nagios agent if we want, make a clever backup script and solve the space issue by reinstalling everything. But if we go the pain of reinstallation and update, a more standard setup would be cleaner and easier in the future, by using straight tarball from upstream, by using standard system to cache the data, etc, etc. I can get you a publicly accessible vm on phx lab for the migration. but the DNS will take some time to change itself, that would suffice for you? I do not think we should migrate before testing a bit, so we can in the mean time reduce the DNS ttl, and it should be good once the migration is done. If so, tell me the OS you need and how much space you tihnk we will need for it. Either we go on el6, or el7. I personally prefer el7 due to newer features, but the main concern i would have is with puppet on it, and if that's fully ok with the current setup. And I think having 40G would be largely enough. We have 10G now, so with 40G, we have space for backup. And we can increase the size of the disk at will, no ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Cannot loging to ovirt wiki
Le mercredi 17 septembre 2014 à 11:27 -0400, Dave Neary a écrit : Hi, On 09/17/2014 11:07 AM, David Caro wrote: On Wednesday, September 17, 2014 10:56:58 AM Dave Neary wrote: Hi, I've documented the issue in the past: http://lists.ovirt.org/pipermail/infra/2013-October/004183.html It would be possible to migrate the database to the separate gear in the ovirt tenant - http://mysqlserver-ovirt.rhcloud.com/ - and use that database with the appropriate environment variables on the wiki gear. Nice! I can do all that only with ssh access? (I have to the wiki, but not sure if I have it to the mysqlserver-ovirt) I guess so, although for something delicate like that I was planning to let misc take it on. As I said in the past, the plan wouldn't work. To have 2 gears communicate, we need to have them setup in a specific way, not just 2 gears in the same account. If one is moved to another node, we need to have a specific triggers on the webserver gear to trigger a potential configuration change. More over, this could mean having more patching to do on mediawiki side, which mean forking. Unless someone step to maintain a mediawiki fork, this is not gonna happen. I still maintain that the solution is to use openshift for what it is currently ( may be different once we have openshift v3 ), a Paas where you develop the software, not one where yo take out of the shelf software. The current mediawiki installation is already outdated when it come to security, so it should be upgraded anyway. It would be easier for me to just have a VM and use apache + tarball than openshift. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Can't log in, wiki complains that cookies are disabled
Le lundi 15 septembre 2014 à 12:06 +0200, David Caro a écrit : Should be fixed, can you confirm? I guess disk full ? this also corrupted the search index. To solve, connect on the gear, and run mysql repair thetable , on mysql prompt ( replace the table with corrupted table ). On Monday, September 15, 2014 11:48:21 AM Juan Hernandez wrote: Hello, I'm getting the following error when trying to login to the wiki: Login error oVirt_Wiki uses cookies to log in users. You have cookies disabled. Please enable them and try again. I have cookies enabled, and tried with two different browsers. Looks like this happened in the past, and it is caused when disk space is exhausted. Can you take a look? Thanks in advance, Juan Hernandez ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Wiki search doesn't work
Le mercredi 03 septembre 2014 à 09:09 -0400, Omer Frenkel a écrit : trying to search for a page on the wiki produce this err (browsing directly to the desired page worked): not sure if related to login issues from yesterday, because i was able to login. Database error A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was: (SQL query hidden) from within function SearchMySQL::searchInternal. Database returned error 145: Table './openshift_mediawiki/mw_searchindex' is marked as crashed and should be repaired (127.7.52.129). Looking at repairing it. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Wiki search doesn't work
Le mercredi 03 septembre 2014 à 15:20 +0200, Michael Scherer a écrit : Le mercredi 03 septembre 2014 à 09:09 -0400, Omer Frenkel a écrit : trying to search for a page on the wiki produce this err (browsing directly to the desired page worked): not sure if related to login issues from yesterday, because i was able to login. Database error A database query syntax error has occurred. This may indicate a bug in the software. The last attempted database query was: (SQL query hidden) from within function SearchMySQL::searchInternal. Database returned error 145: Table './openshift_mediawiki/mw_searchindex' is marked as crashed and should be repaired (127.7.52.129). Looking at repairing it. Fixed it seems, please report if anything is not working. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: oVirt wiki login issues
Le lundi 01 septembre 2014 à 11:43 +0200, David Caro a écrit : On 09/01, Moti Asayag wrote: Hi, Seems like [1] happens again: Login error oVirt_Wiki uses cookies to log in users. You have cookies disabled. Please enable them and try again. Could you please handle it ? Same issue as last time, I see that the database is occupying most of the space, I see 2 possible solutions (can be applied together): * Shrink database: - Regenerating the ibdata: This requires dumping, deleting and recreating the database to regenerate the ibdata file. We can also separate that file per table during the process. - Cleaning up temporary and unnecessary data from the database: This requires knowledge of the internal structure of the database of the wiki, anyone has knowledge on that? * Expand space: - Not sure how to proceed here, but we can ask for a bigger gear to hold the wiki, we are using 8GB now. In any case, bkp, misc, any ideas? in the meantime, I removed logs, and copied them offline, this give us 400 M. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: oVirt wiki login issues
Le mardi 02 septembre 2014 à 18:17 +0200, Michael Scherer a écrit : Le lundi 01 septembre 2014 à 11:43 +0200, David Caro a écrit : On 09/01, Moti Asayag wrote: Hi, Seems like [1] happens again: Login error oVirt_Wiki uses cookies to log in users. You have cookies disabled. Please enable them and try again. Could you please handle it ? Same issue as last time, I see that the database is occupying most of the space, I see 2 possible solutions (can be applied together): * Shrink database: - Regenerating the ibdata: This requires dumping, deleting and recreating the database to regenerate the ibdata file. We can also separate that file per table during the process. - Cleaning up temporary and unnecessary data from the database: This requires knowledge of the internal structure of the database of the wiki, anyone has knowledge on that? * Expand space: - Not sure how to proceed here, but we can ask for a bigger gear to hold the wiki, we are using 8GB now. In any case, bkp, misc, any ideas? in the meantime, I removed logs, and copied them offline, this give us 400 M. I would love to say that it took me 3h to restart the wiki, but the reality is more someone came and asked me a urgent question and I forgot to restart the wiki after the interruption (and went back home) :/ So now, it should be working fine. I still recommend to migrate it out of openshift since this would make the maintenance a bit easier, but we need to have the lab ready. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: oVirt wiki login issues
Le lundi 01 septembre 2014 à 11:43 +0200, David Caro a écrit : On 09/01, Moti Asayag wrote: Hi, Seems like [1] happens again: Login error oVirt_Wiki uses cookies to log in users. You have cookies disabled. Please enable them and try again. Could you please handle it ? Same issue as last time, I see that the database is occupying most of the space, I see 2 possible solutions (can be applied together): * Shrink database: - Regenerating the ibdata: This requires dumping, deleting and recreating the database to regenerate the ibdata file. We can also separate that file per table during the process. - Cleaning up temporary and unnecessary data from the database: This requires knowledge of the internal structure of the database of the wiki, anyone has knowledge on that? * Expand space: - Not sure how to proceed here, but we can ask for a bigger gear to hold the wiki, we are using 8GB now. In any case, bkp, misc, any ideas? I would rather migrate the wiki out of openshift when we can have a server for that in phx2. We will no longer be constrained by hosting, and we will be able to administer it in a more conventional way. Openshift is made to deploy software that you deploy and write yourself, not really to host stuff made by others who are not made for this. We shoehorn mediawiki on it, and we start to see the limit of the approach :/ Migrating to a set of 2 gears would only work for scaled gears, which mean reinstalling the whole setup almost from scratch. So I would prefer, if we need to do that, to move to a regular VM ( where then we can enable cache, varnish, etc to make things faster ) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Can't login into ovirt wiki anymore
Le mercredi 27 août 2014 à 03:52 -0400, Eyal Edri a écrit : - Original Message - From: David Caro dcaro...@redhat.com To: Sandro Bonazzola sbona...@redhat.com Cc: infra infra@ovirt.org Sent: Wednesday, August 27, 2014 10:42:41 AM Subject: Re: Can't login into ovirt wiki anymore On 08/27, Sandro Bonazzola wrote: Hi, today looks like I can't login anymore on ovirt wiki. Nothing has changed on my system configuration but now I have: Login error oVirt_Wiki uses cookies to log in users. You have cookies disabled. Please enable them and try again. Anybody else is experiencing this issue? I'm not sure it's client side. I have the same issue, and yesterday Brian Proffit said he had that issue too :/ Anyone with openshift knowledge around? is there a system with can open ticket on this to openshift? not as I know. But brian did face the issue tomorow and it solved by itself before I could investigate. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Lists.ovirt.org full again
Hi, FYI, lists.ovirt.org was full again, and I had the nice surprise to see 60 errors messages in my mailbox during the night. I didn't had time to look why ( besides ovirt 3.5 ), and cannot devote much time to it, being at GUADEC this week and at Flock next one. I just removed various stuff, freeing around 300 to 400M, which would be more than enough for mailman/postfix to work. -- Michael Scherer ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: upgrade of alterway02 ovirt node
Le dimanche 06 juillet 2014 à 11:45 -0400, Eyal Edri a écrit : great to hear. could you share some info on the upgrade (was it 3.2 - 3.4?) yes maybe there are some bugs ovirt need to fix or flows we need to be aware of them? well, I never did a upgrade, so I did it wrong, but ewoud fixed everything. Now, I have looking on how I can start the VM with a iso, but the iso domain is not started and so I cannot boot on it and so I am reading a bit of doc before bothering people with question :) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
upgrade of alterway02 ovirt node
Hi, prior to migrating mailling list to a new VM, Ewoud and I did upgrade ovirt engine on alterway02 to 3.4. It didn't really went as planned ( between problematic upgrade and left over of file in /var/tmp, some conflict in the rpm upgrade and the upgrade script not working as expected that Ewoud fixed ). Now, it should be good, so I will proceed tomorrow to migrate the data, and then do the test to migrate as soon as possible. I will keep people updated. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Anacron job 'cron.daily' on linode01.ovirt.org
Le mercredi 02 juillet 2014 à 04:40 -0400, Anacron a écrit : /etc/cron.daily/0logwatch: gzip: stdout: No space left on device system zcat failed: 256 at /usr/sbin/logwatch line 896. So, some cleaning was done this morning by Eyal and Ewoud on the server, and by me yesterday to get a few meg. We are now at 2.4G, but discussing with Eyal on irc, we proposed to move the mailling list ( ie archive and stuff ) to a VM on alterway ovirt installation. This may be disruptive, and since that's not in puppet, it may take longer than I wish. So I propose to install the services, but not declare them as being MX, do some testing. Then once the lists are on the new list server, stop mailman on the current server, put it as secondary MX put the first one as primary MX and migrate over. This should free 4G, but more importantly, this would split the download server from the ml, which mean that download server being full will not stop communication around the project anymore. Any comment on this ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: [URGENT] not enough space for releasing 3.5.0 beta.
Le vendredi 27 juin 2014 à 09:53 +0200, Sandro Bonazzola a écrit : Il 27/06/2014 09:13, logwa...@linode01.ovirt.org ha scritto: - Disk Space Begin Filesystem Size Used Avail Use% Mounted on /dev/xvda97G 95G 1.4G 99% / /dev/xvda = 99% Used. Warning. Disk Filling up. After feeling the server and that I got alert, I did clean and remove 1G of data ( ie, the iso i spoke about on my mail last month ). I copied it to stats.ovirt.org So now, it is officially critical to get a new server for ml and download. Do we have ressource left at rackspace or somewhere, or should I try to find a temporary place until we get something somewhere ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Proposal for hosting our own set of modules
Le mardi 24 juin 2014 à 09:59 +0200, Ewoud Kohl van Wijngaarden a écrit : On Mon, Jun 23, 2014 at 06:02:51PM +0200, Michael Scherer wrote: Le lundi 23 juin 2014 à 16:42 +0200, Ewoud Kohl van Wijngaarden a écrit : On Mon, Jun 23, 2014 at 04:30:55PM +0200, Michael Scherer wrote: ( https://github.com/adrienthebo/r10k ) , it requires use to have the modules/ directory to be managed by librarian-puppet. In turn we need to have the modules in git, we can address them by path: mod puppetlabs/apt, :git = git://github.com/fake/puppet-modules.git, :path = modules/apt This wouldn't requires to change much, besides adding the module to Puppetfile and creating a git repository. If no one disagree, I will request the git repository. ( in the mean time, i did create a sample awstats repository for stats.ovirt.org, so we can have something to push ) Unless we plan to make them reusable for other projects, I don't see the benefit. If we do plan to make them reusable, we should IMHO also publish them on the forge. Another potential issue is how we decide when to deploy. We could have a specific commit ID and update our Puppetfile every time, but again, little benefit over having them in one tree. I was under the impression you could just give a tag and so use master ? ( and using branch for development ) You can, but how do we know we want to update and to which version? branches can easily break compatibility and it will bite you at some point. It depend on the release model of our own modules. We can have a policy of master should be deployable and forever retro compatible but once expressed like this, it doesn't sound sustainable (even if that's what we would do with a single git repository ). In case you're unaware, we already load modules/* (which is managed by r10k) and site/* (tracked in git). That means we can host our modules in site and spit off when they are reusable. It was not very obvious that site/* was for modules, indeed :) But my fault, I should also have read the doc in the git repository who clearly say that. I must also say that using r10k for different environments seems a bit overkill, as we seem to only have 1 single environment anyway ( but I am not using r10k usually, as I do not have enough systems for that and write all stuff myself ). The alternative is either include them or git submodules. I'm not a fan of the former and I've seen people have issues with the latter where people accidentally commit an older version. It's also hard to spot in reviews because you only see two git commit IDs with no idea if it's an upgrade or downgrade. That's why a Puppetfile-based solution was chosen. I am not fan of git submodule either. But the Puppetfile file part is good, it just the multiple env that I found weird. Now, if that's unavoidable, no problem. Minor nitpick: technically we have 2 environments and we should make an effort to remove master. This is done by changing the default branch to production and then remove master. The only way I know is logging into gerrit through SSH and changing infra-puppet.git/HEAD to point to production instead of master. Then you can use the gerrit UI to delete the master branch. As we discussed on IRC later, I would be +1 for this, provided that this doesn't break too much assumption in docs ( especially gerrit doc around ). But as we discussed too, maybe we should just push to use git-review. And if we do plan on creating modules repos, I'd be in favor of having one git repo per puppet module since that is what most people would expect. One git repo per module is a bit annoying when we are planning to write a lot of modules. But I guess it all depend on the time it take to create one. I would definitely prefer a approach of 1 big git as long as we are growing fast and maybe split later, since it permit faster growth ? That's exactly what I was thinking. This also bring the question of using external module vs using our own. I am more the kind of guy that write his own (while I am not the kind of guy who write his own code), mostly because I started to use puppet before the forge was up, and because I did have very precise ideas on what I wanted to achieve, but I am not against using stuff if that's the best practices :) I would however make sure we have proper guidelines on when we take a module, as I am not sure they are all equally good :/ -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Puppet modules organization
Le mardi 17 juin 2014 à 11:36 +0200, David Caro a écrit : Hi everyone! I'm starting a thread to discuss the puppet modules organization. There are two proposed ways of organizing them: 1.- Using a unique module named ovirt_infra 2.- Using multiple modules, named ovirt_* Feel free to propose other alternatives, the main points for each one are: 1.- Everything inside one module, easy to find 1.- Easy to add a new class, just create the file 1.- Easy to create hard to maintain code 1.- Easy to create very interdependent code 2.- Enforces modularization of the different code (one module, one task), that brings 2.- Easier to test 2.- Safe to reuse 2.- More organized (not everything in the same place) 2.- It's the most common way of organizing puppet manifests, so the main guidelines, patterns and most of the documentation expects this way of working. Please send your comments and if too many I'll open a pad with the them for easy review. I vote for #2, modularized organization. Everything would still be in the same git, just to be clear ? I would vote for #2, and in fact, I would even as far as separating the ovirt specific stuff from the functionnal module. But this can be done later. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Proposal for hosting our own set of modules
Hi, so following the discussion on puppet module, I would propose that we create a new infra-puppet-modules module on gerrit. The division between this module and the current one would be like this : infra-puppet would hold the manifests, the site, the Puppetfile , etc infra-puppet-modules would hold directories, 1 per module we develop (not the external one we use, they would still be on github or anywhere, pulled by librarian-puppet) . Since we are using librarian-puppet ( https://github.com/rodjek/librarian-puppet ) and r10k ( https://github.com/adrienthebo/r10k ) , it requires use to have the modules/ directory to be managed by librarian-puppet. In turn we need to have the modules in git, we can address them by path: mod puppetlabs/apt, :git = git://github.com/fake/puppet-modules.git, :path = modules/apt This wouldn't requires to change much, besides adding the module to Puppetfile and creating a git repository. If no one disagree, I will request the git repository. ( in the mean time, i did create a sample awstats repository for stats.ovirt.org, so we can have something to push ) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Proposal for hosting our own set of modules
Le lundi 23 juin 2014 à 16:42 +0200, Ewoud Kohl van Wijngaarden a écrit : On Mon, Jun 23, 2014 at 04:30:55PM +0200, Michael Scherer wrote: Hi, so following the discussion on puppet module, I would propose that we create a new infra-puppet-modules module on gerrit. The division between this module and the current one would be like this : infra-puppet would hold the manifests, the site, the Puppetfile , etc infra-puppet-modules would hold directories, 1 per module we develop (not the external one we use, they would still be on github or anywhere, pulled by librarian-puppet) . Since we are using librarian-puppet ( https://github.com/rodjek/librarian-puppet ) and r10k We're only using r10k, not librarian-puppet. Indeed. But in the end, we still use Puppetfile nonetheless, no ? ( https://github.com/adrienthebo/r10k ) , it requires use to have the modules/ directory to be managed by librarian-puppet. In turn we need to have the modules in git, we can address them by path: mod puppetlabs/apt, :git = git://github.com/fake/puppet-modules.git, :path = modules/apt This wouldn't requires to change much, besides adding the module to Puppetfile and creating a git repository. If no one disagree, I will request the git repository. ( in the mean time, i did create a sample awstats repository for stats.ovirt.org, so we can have something to push ) Unless we plan to make them reusable for other projects, I don't see the benefit. If we do plan to make them reusable, we should IMHO also publish them on the forge. Another potential issue is how we decide when to deploy. We could have a specific commit ID and update our Puppetfile every time, but again, little benefit over having them in one tree. I was under the impression you could just give a tag and so use master ? ( and using branch for development ) In case you're unaware, we already load modules/* (which is managed by r10k) and site/* (tracked in git). That means we can host our modules in site and spit off when they are reusable. It was not very obvious that site/* was for modules, indeed :) But my fault, I should also have read the doc in the git repository who clearly say that. I must also say that using r10k for different environments seems a bit overkill, as we seem to only have 1 single environment anyway ( but I am not using r10k usually, as I do not have enough systems for that and write all stuff myself ). And if we do plan on creating modules repos, I'd be in favor of having one git repo per puppet module since that is what most people would expect. One git repo per module is a bit annoying when we are planning to write a lot of modules. But I guess it all depend on the time it take to create one. I would definitely prefer a approach of 1 big git as long as we are growing fast and maybe split later, since it permit faster growth ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Minutes :: oVirt infra weekly :: 2014-06-16
Le lundi 16 juin 2014 à 17:44 +0200, David Caro a écrit : === #ovirt: oVirt Infra === Meeting started by dcaro at 14:22:51 UTC. The full logs are available at http://ovirt.org/meetings/ovirt/2014/ovirt.2014-06-16-14.22.log.html . Meeting summary --- * Hosting (dcaro, 14:29:51) * new lab in phx is almost here, found some issues installing base os (dcaro, 14:30:36) * ACTION: create a pad with the current list of hosts to add to foreman (dcaro, 14:41:43) So I created a pad: http://etherpad.ovirt.org/p/service_list_ovirt I am gonna try to complete it until end of week, but as I got pulled for others tasks around, feel free to complete it. Then once we have a up to date working document, and once we decided what information we need, we can place it on the wiki. I did also use my internal access to the RH DNS to get a list of the current dns names, and I have a few questions on some of them: - ftp.ovirt.org point to resources.ovirt.org; yet, there is no ftp server running there. Can we clean this name ( after proper announce, in case people used that for yum access ) - puppet.ovirt.org point to the same server, but the puppet master is on foreman, different server. This one should be removed IMHO, as it prevent automated configuration in puppet (to some extend ), and can be confusing. - plugin/plugins.ovirt.org , nothing use it from a quick 30 seconds search on Google. Do we need to keep that ? - glance.ovirt.org. Only ssh is running on that server, do we need to keep it ? ( in fact, is this the proper server ? ) I can proceed to remove them if we do not use them ( just to have a cleaner zone file ). -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Followup on today lists.ovirt.org http outage
Hi, Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck. So as I am more of the kind shoot first, ask later, I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP. in short : - yum install mod_limitipconn - add IfModule mod_limitipconn.c MaxConnPerIP 10 /IfModule to /etc/httpd/conf.d/resources.ovirt.org.conf I guess we should add this in some puppet module somewhere ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Followup on today lists.ovirt.org http outage
Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a écrit : On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote: Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck. So as I am more of the kind shoot first, ask later, I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP. I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README. Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ). Even if a solution that requires no maintainance is maybe a better solution for now. in short : - yum install mod_limitipconn - add IfModule mod_limitipconn.c MaxConnPerIP 10 /IfModule to /etc/httpd/conf.d/resources.ovirt.org.conf I guess we should add this in some puppet module somewhere ? We should, but the whole apache config isn't puppetized yet. I've been slacking on that because we want to move away from that server, but maybe we should bite the bullet and do it on the current server. Yep, and I think it would be easier to move away from the server if it is in puppet :) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Followup on today lists.ovirt.org http outage
Le mardi 17 juin 2014 à 16:17 +0200, Sandro Bonazzola a écrit : Il 17/06/2014 16:13, Michael Scherer ha scritto: Le mardi 17 juin 2014 à 15:55 +0200, Ewoud Kohl van Wijngaarden a écrit : On Tue, Jun 17, 2014 at 03:47:14PM +0200, Michael Scherer wrote: Brian pinged me on a failure on lists.ovirt.org around 13h15 UTC. After scratching my head for a while ( since everything was running fine, despites regular Out of memory on the server ), it turned out to be a user trying to get the iso with a download accelerator. I first added more server, but without luck. So as I am more of the kind shoot first, ask later, I did kill the connexion with iptables, then limit it with iptables ( but with some side effect ), then installed mod_limitipconn to limit to 10 tcp connexion per IP. I'm all in favor of this. Maybe we should mention we have mirrors with MUCH more bandwith in our README. Or maybe we do not need to tell that to people and use a redirector ? ( like mirrorbrain, etc ). This won't allow to download packages until mirrors are synced. Now yum repo files have mirrorlist pointing to mirrors and baseurl pointing to ovirt.org. Introducing automated redirection won't allow this anymore. It work quite fine for fedora. But we indeed to make sure this doesn't break stuff for people, especially older setup. I guess we also want to still get some download stats. But we do not have much issues or risk with rpms, more on iso download, as they take more bandwidth and for longer time, and I think there is more risk of having a download accelerator for that. So what about just make a redirect for iso (if possible, something smart enough to not redirect if the mirror is not synced), and keep rpms as is ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: stats.ovirt.org ?
Le lundi 16 juin 2014 à 13:38 +0200, David Caro a écrit : On Mon 16 Jun 2014 11:40:10 AM CEST, Michael Scherer wrote: Le mercredi 11 juin 2014 à 19:17 +0200, David Caro a écrit : On Wed 11 Jun 2014 06:45:27 PM CEST, Michael Scherer wrote: Hi, what is the server stats.ovirt.org supposed to do ? It seems to be unconfigured, and the carbon setup for graphing do not seems completely functionnal ( ie not configured to start on boot, carbon-relay is wrong, and do not seems to be configured by puppet ). It also run Fedora 18, so should be upgraded. ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra That server was meant to hold some statistics and performance graphing for the infra, I started configuring it but never finished, just got webalizer running there and publishing the stats about resources.ovirt.org, gerrit linode01 and lists web servers. The idea was to get graphite working there and send stats from nagios to it. But also install statsd (or similar) on the machines to have more grained stats if needed or other custom statistics or events (for example patches merged, patches created, etc.). Also, it seems that graphite is not very actively maintained: https://github.com/graphite-project/ The commits graphs look a bit flat :/ Yep, seems that it got a little quieter lately, it's a great tool though, and there are a lot of tools around it (specially front-ends). I think it's better than cacti and a lot easier to automate when needed. So what do we do with the VM ? Update it to Fedora 20, drop it ? We should update it and properly configure it, but seen that it¡s almost not configured, we might want to start from a fresh vm. It still has the awstats setup on it, so in any case we should not delete it. Ok, so I will at least try to upgrade to F19, likely F20. As it is not really in production, I guess I can do that in the week ? Also, awstats do not seems to be in puppet, and I think it should, do we have a process for adding a module to git and puppet ? (ie, adding a module we wrote rather than one of the forge) -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Followup on yesterday ovirt.org outage
Hi, so yesterday, www.ovirt.org was down for a few minutes around 13h45 UTC. As a few people did ask on irc, I did a followup with openshift team. It turn out to have been just a security fix reboot, and that the wiki was planned to be restarted. I am not sure if someone did receive a alert or something, but I guess I will ask for it next time ( or maybe follow the proper ml/twitter/etc ). -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
stats.ovirt.org ?
Hi, what is the server stats.ovirt.org supposed to do ? It seems to be unconfigured, and the carbon setup for graphing do not seems completely functionnal ( ie not configured to start on boot, carbon-relay is wrong, and do not seems to be configured by puppet ). It also run Fedora 18, so should be upgraded. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Selinux, because it is friday
Le lundi 09 juin 2014 à 13:19 +0200, Michael Scherer a écrit : Le dimanche 08 juin 2014 à 02:47 -0400, Eyal Edri a écrit : - Original Message - From: David Caro dcaro...@redhat.com To: Michael Scherer msche...@redhat.com Cc: infra@ovirt.org Sent: Friday, June 6, 2014 5:24:20 PM Subject: Re: Selinux, because it is friday On Fri 06 Jun 2014 04:06:00 PM CEST, Michael Scherer wrote: Hi again, while looking at servers, I also couldn't help noticing that selinux is either disabled or set as permissive on the few servers I looked, one even having auditd disabled. So I did enable auditd with the goal of collecting violation in audit.log ( aka AVC ), and I plan to look at them. I already started to fix a few violations showing up in the log. Sometime, this would just be enabling a boolean to configure selinux ( ie, enable some specific access ), sometime, it was just wrongly labelled file ( on monitoring.ovirt, mostly ). I do not plan to set selinux in enforcing mode before having check that there is no problem for a longer period of time, and of course, not if people think it is not wise. I also so far only propose to do that host by host, as I guess the jenkins ones may be more complex to limit. I wil report with what I foud and so we will discuss if we make the switch or not. thanks for this effort michael! security is always important and sometimes unfourtunately gets pushed behind other urgents tasks. after we've made sure enabling selinux doesn't break anything, can we ensure its set for all servers via puppet? yes. Either by forcing the content of /etc/selinux/config, or with augeas. I would even be more radical and make sure selinux is set to enforcing with nagios i.e. get a alert if someone/something disable it. also - might worth opening a ticket in trac on it for tracking progress.. yep, good point. https://fedorahosted.org/ovirt/ticket/158 I am completing the ticket with what we discuss -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
infra security update
Hi, Due to CVE on openssl and on kernel, I did upgrade various piece of the infrastructure ( foreman, lists, stats, monitoring ), which implied a few reboots ( due to kernel lagging behind, which is not that great with local root exploit ). As this is friday and I assumed most of the Tel Aviv office was not working, i hope this kept the disruption to a minimum. However, if something is broken, please tell it so we can fix. This also got me thinking. In order to bring a bit more order, what about having a fixed schedule for upgrade ? In my previous position, we were doing that once per month ( except during end of quarter freeze ), with mandatory reboot ( cause if something do not boot, you want to know it when you have a planned outage, not when everyone is running around updating stuff ). Fedora has a rather complex procedure to decide what to upgrade, hilighted on http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt So we could adopt a schedule ( once per month, unless there is something critical, in which case we do it ASAP, with warning on the list and irc ). The schedule should of course take in account business need, which is release schedule of ovirt. So what about first friday of the month, unless exception ? And by update, i mean yum upgrade -y. Cleaning the list of repo on various servers is also IMHO another task to discuss, to make sure the task can be safely executed. ( having something like mcollective/ansible/func is also needed, but that's more a convenience than a requirement at this stage ). -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Selinux, because it is friday
Hi again, while looking at servers, I also couldn't help noticing that selinux is either disabled or set as permissive on the few servers I looked, one even having auditd disabled. So I did enable auditd with the goal of collecting violation in audit.log ( aka AVC ), and I plan to look at them. I already started to fix a few violations showing up in the log. Sometime, this would just be enabling a boolean to configure selinux ( ie, enable some specific access ), sometime, it was just wrongly labelled file ( on monitoring.ovirt, mostly ). I do not plan to set selinux in enforcing mode before having check that there is no problem for a longer period of time, and of course, not if people think it is not wise. I also so far only propose to do that host by host, as I guess the jenkins ones may be more complex to limit. I wil report with what I foud and so we will discuss if we make the switch or not. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: infra security update
Le vendredi 06 juin 2014 à 14:36 +0200, Ewoud Kohl van Wijngaarden a écrit : On Fri, Jun 06, 2014 at 01:29:44PM +0200, Michael Scherer wrote: Due to CVE on openssl and on kernel, I did upgrade various piece of the infrastructure ( foreman, lists, stats, monitoring ), which implied a few reboots ( due to kernel lagging behind, which is not that great with local root exploit ). As this is friday and I assumed most of the Tel Aviv office was not working, i hope this kept the disruption to a minimum. However, if something is broken, please tell it so we can fix. Nice work. This also got me thinking. In order to bring a bit more order, what about having a fixed schedule for upgrade ? In my previous position, we were doing that once per month ( except during end of quarter freeze ), with mandatory reboot ( cause if something do not boot, you want to know it when you have a planned outage, not when everyone is running around updating stuff ). Fedora has a rather complex procedure to decide what to upgrade, hilighted on http://infrastructure.fedoraproject.org/infra/docs/massupgrade.txt At my previous employer we had something similar. I also wrote a puppet custom fact reboot_needed which checks if the running kernel matches the default kernel that would be booted. We also want to restart services that are affected. Hence a reboot would be simpler from a engineering point of view. So we could adopt a schedule ( once per month, unless there is something critical, in which case we do it ASAP, with warning on the list and irc ). +1 The schedule should of course take in account business need, which is release schedule of ovirt. So what about first friday of the month, unless exception ? We should also make sure that we don't reboot ALL servers at once. So if we have multiple centos 6 jenkins slaves, try to just reboot one at a time. Also would be nice if the slave did proper scheduling in jenkins so no jobs are running. Yep, proper orchestration would be nice. Now, if jenkins is resilient enough ( ie, if it can survive a 5 minutes downtime ), it may not be that business critical to have it running all the time. I am not a jenkins expert, not even a user, so I defer to David for that. And by update, i mean yum upgrade -y. Cleaning the list of repo on various servers is also IMHO another task to discuss, to make sure the task can be safely executed. ( having something like mcollective/ansible/func is also needed, but that's more a convenience than a requirement at this stage ). We sometimes have pinned versions on jenkins build slaves. That means we should either do a proper yum versionlock or find something else. Note that I'm all in favor of being able to to a blind yum upgrades. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Fwd: [ovirt-users] Trying to reset password for ovirt wiki
Le jeudi 29 mai 2014 à 22:12 -0400, Dave Neary a écrit : Hi, Do we have an issue with the mail server for the oVirt wiki? Who can take a look at this? So, it seems that ovirt wiki is configured to use the ovirt list server as a relay, but the only authentication used is by ip address. And as the ip of www.ovirt.org is not guaranteed to be stable ( due to openshift nature, node can move around on EC2 ), this broke. The short term solution is to change the ip on lists.ovirt.org to use the current one. It seems to have been done more than once. The long term solution is to use proper password auth to be able to relay. I implement quickly the first one. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Using the free space on rackspace for old releases
Le mercredi 21 mai 2014 à 07:40 -0700, Karsten Wade a écrit : On 05/21/2014 12:54 AM, David Caro wrote: Hi! We are having some issues on linode due to space limitations, and as we are not using the space we have on rackspace, maybe we can move there the old releases. That will require configuring apache on the rackspace node that will host the files and also some configuration on linode, but not complicated as far as I can tell. Any objections or other ideas? Feel free to move everything over. :) Actually, why not move all of resources.ovirt.org, leaving Mailman for a separate migration to a new, different server? I'd like to decommission that VM as soon as we can. So I looked around at the existing ressources. Alterway ovirt cluster still has 4G of memory, so we could create 1 VM there for mailman ( 1g should be enough, even if I suspect spamassassin may be more confortable with more ). The hosts in rackspace have a lot more free memory, so we could also move the VM there. The only blocking point would be to get enough IP address ( ie, 1 would be enough ). -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Lists.ovirt.org is full ( or almost )
Hi, I just started to receive mail from postfix about insufficient storage on lists.ovirt.org. So this would have blocked all ML. I did some cleaning in urgency ( yum cache, etc ), but we would still need to remove urgently lots of space to avoid seeing the issue again next week. So I would like to propose the following : - short term - remove the iso in /home/obasan , that's ovirt 3.3, we are at 3.4, so maybe not needed. that's 1G. - mid term - do backup somewhere else than on this server. If the server crash, having backup on it will be useless :) ( ie, the 9G of gerritt backup ) - have smarter backups. Doing a full backup of gerrit mean there is lots of duplication. Not sure if we should backup tmp file, for example. - remove texlive. it take half a gigabit, only for one version of mediawiki. Which is weird on this server since it is not installed, IIRC - longer term - move to a bigger server - have monitoring of the root partition usage so this can be detected before it become a issue - try to have less automated mails sent on ml, as they take space (logwatch for one, but maybe others) Any opinions ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: Lists.ovirt.org is full ( or almost )
Le mercredi 21 mai 2014 à 02:17 +0200, Michael Scherer a écrit : Hi, I just started to receive mail from postfix about insufficient storage on lists.ovirt.org. So this would have blocked all ML. I did some cleaning in urgency ( yum cache, etc ), but we would still need to remove urgently lots of space to avoid seeing the issue again next week. So I would like to propose the following : - short term - remove the iso in /home/obasan , that's ovirt 3.3, we are at 3.4, so maybe not needed. that's 1G. - mid term - do backup somewhere else than on this server. If the server crash, having backup on it will be useless :) ( ie, the 9G of gerritt backup ) ok, turn out that's already offline backup. My fault, shouldn't try to think 2h past midnight after trying to fix stuff :) - have smarter backups. Doing a full backup of gerrit mean there is lots of duplication. Not sure if we should backup tmp file, for example. - remove texlive. it take half a gigabit, only for one version of mediawiki. Which is weird on this server since it is not installed, IIRC - longer term - move to a bigger server - have monitoring of the root partition usage so this can be detected before it become a issue - try to have less automated mails sent on ml, as they take space (logwatch for one, but maybe others) Any opinions ? ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
FYI, node-devel and devel lists got merged
Hi, Brian did ask me to merge the node-devel@ and devel@ list, which I just did. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
No privacy policy ?
Hi, People contacted me about mailling list issue, but as I think there is no reason to not discuss with the community at large, I prefer follow here. It seems that for some reason, we are being blocked by microsoft.com due to autoamted spam detection. I did look at https://mail.live.com/mail/troubleshooting.aspx and basically, the ip is listed Mail rejected by Outlook.com for policy reasons. Reasons for rejection may be related to content with spam-like characteristics or IP/domain reputation. If you are not an email/network admin please contact your Email/Internet Service Provider for help. Looking on the web, i found http://aacable.wordpress.com/2013/03/26/howto-remove-your-mail-server-ip-from-hotmail-black-list/ and indeed, I confirmed that's the IP of lists.ovirt.org who is blocked. We can request to be delisted ( using https://support.live.com/eform.aspx?productKey=edfsmsbl3ct=eformtsscrx=1 ), but that requires to open a account at Microsoft ( which I did open ), and several informations. Among them, they ask for a privacy policy. And it turn out we don't have one : http://www.ovirt.org/oVirt_Wiki:Privacy_policy So while I can continue without it ( or hoping no one will check ), I would be more confortable to ask for being delisted with one. Is someone volunteer to tackle that task ? -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: oVirt Site Concerns
Le mardi 08 avril 2014 à 08:24 -0400, Brian Proffitt a écrit : All: Earlier this morning, I was pulling in the stats from Bitergia, and got this message: Gear 847edb45aea84198838f915be6faa066 is using 91.2% of disk quota I know updating the size of the gear was something on the list of things to do, but I was under the impression we'd updated the size of the gear. On investigation, Michael Scherer discovered that the mysql database is taking up 5.9 GB, and the logs 2 GB. Is there anything we can do to mitigate the size of these instances? We can erase the logs. We can also try to see if the db can be shrinked, but I am not sure it can be done. Also, you have probably heard about the HeartBleed OpenSSL vulnerability [1], but just in case, we need to make sure this has been addressed. OpenShift should be fine, but any service using TLS needs to be updated ASAP. Michael has already checked, and puppetmaster and lists.ovirt.org appear to be okay. That's ok on lists because there is no tls at all. And I just checked the foreman https with a small reproducer, better update anyway. -- Michael Scherer Open Source and Standards, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
Re: oVirt.org down
Le mardi 04 mars 2014 à 06:46 -0500, Brian Proffitt a écrit : The site is down now, and was down earlier this morning for about 48 minutes. seems to be back now. -- Michael Scherer Open Source and Standard, Sysadmin signature.asc Description: This is a digitally signed message part ___ Infra mailing list Infra@ovirt.org http://lists.ovirt.org/mailman/listinfo/infra
