[Openstack] wsgi code duplication
Hi Everyone, i've been looking through wsgi code, and i have found a lot of duplicated code between all the projects. Running a quick and dirty search, i get the following numbers (just focusing on glance horizon keystone nova openstack-common quantum swift projects): - There are 87 classes defined, but only 40 different. Rough numbers: (Number of times the class appears) 6 Request 5 Server 5 Router 5 Middleware 5 Debug 4 WritableLogger 3 XMLDictSerializer 3 XMLDeserializer 3 TextDeserializer 3 Resource 3 JSONDictSerializer 3 JSONDeserializer 3 DictSerializer 3 Application 3 ActionDispatcher 2 ResponseSerializer 2 RequestHeadersDeserializer 2 RequestDeserializer 2 Fault 2 Controller And some of them only defined and used once: (some are the same with different name like ResponseHeader*s*Serializer and ResponseHeaderSerializer) 1 WSGIContext 1 Serializer 1 ResponseObject 1 ResponseHeadersSerializer 1 ResponseHeaderSerializer 1 ResourceExceptionHandler 1 Resource 1 OverLimitFault 1 MetadataXMLDeserializer 1 Loader 1 JSONResponseSerializer 1 JSONRequestDeserializer 1 FilterFactory 1 ExtensionRouter 1 ControllerMetaclass 1 ComposingRouter 1 ComposableRouter 1 BasePasteFactory 1 BaseApplication 1 AppFactory Running a code analysis duplication tool like http://clonedigger.sourceforge.net/, the numbers and mostly the same: around 45% of the code is duplicated (A full output is available at http://debostack.org/paste/wsgi.html). I think there are a lot improvements we can get here using openstack-common as the base for all the wsgi code. (As Adam Young reported[1], services can be migrated to use HTTPD, but quantum[2] have some issues. Using the same code, the solution should be the same in all projects). I will open a blueprint for this and update the info as much as possible. [1] - http://adam.younglogic.com/2012/03/keystone-should-move-to-apache-httpd/ [2] - https://lists.launchpad.net/openstack/msg09966.html -- Ghe Rivero *OpenStack Distribution Engineer **www.stackops.com | * ghe.riv...@stackops.com diego.parri...@stackops.com ** | +34 625 63 45 23 | skype:ghe.rivero* * http://www.stackops.com/ * * ADVERTENCIA LEGAL Le informamos, como destinatario de este mensaje, que el correo electrónico y las comunicaciones por medio de Internet no permiten asegurar ni garantizar la confidencialidad de los mensajes transmitidos, así como tampoco su integridad o su correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias. Si no consintiese en la utilización del correo electrónico o de las comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro conocimiento de manera inmediata. Este mensaje va dirigido, de manera exclusiva, a su destinatario y contiene información confidencial y sujeta al secreto profesional, cuya divulgación no está permitida por la ley. En caso de haber recibido este mensaje por error, le rogamos que, de forma inmediata, nos lo comunique mediante correo electrónico remitido a nuestra atención y proceda a su eliminación, así como a la de cualquier documento adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o utilización de este mensaje, o de cualquier documento adjunto al mismo, cualquiera que fuera su finalidad, están prohibidas por la ley. * PRIVILEGED AND CONFIDENTIAL We hereby inform you, as addressee of this message, that e-mail and Internet do not guarantee the confidentiality, nor the completeness or proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L. does not assume any liability for those circumstances. Should you not agree to the use of e-mail or to communications via Internet, you are kindly requested to notify us immediately. This message is intended exclusively for the person to whom it is addressed and contains privileged and confidential information protected from disclosure by law. If you are not the addressee indicated in this message, you should immediately delete it and any attachments and notify the sender by reply e-mail. In such case, you are hereby notified that any dissemination, distribution, copying or use of this message or any attachments, for any purpose, is strictly prohibited by law. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] wsgi code duplication
Hi Ghe, On Tue, 2012-04-24 at 12:15 +0200, Ghe Rivero wrote: Hi Everyone, i've been looking through wsgi code, and i have found a lot of duplicated code between all the projects. Thanks for looking into this. It sounds quite daunting. I wonder could we do this iteratively by extract the code which is most common into openstack-common, move the projects over to that and then start again on the next layer? Cheers, Mark. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] wsgi code duplication
On Apr 24, 2012, at 9:28 AM, Ghe Rivero wrote: On Tue, Apr 24, 2012 at 2:40 PM, Mark McLoughlin mar...@redhat.com wrote: Hi Ghe, On Tue, 2012-04-24 at 12:15 +0200, Ghe Rivero wrote: Hi Everyone, i've been looking through wsgi code, and i have found a lot of duplicated code between all the projects. Thanks for looking into this. It sounds quite daunting. I wonder could we do this iteratively by extract the code which is most common into openstack-common, move the projects over to that and then start again on the next layer? Cheers, Mark. I have plans to try to move as much as possible into openstack-common. I will start with nova as a test bed and see what we get from there. My future plans include db code and tests (in the case of quantum, plugins test also have a lot of duplicated code). I register a bp for the wsgi issue: https://blueprints.launchpad.net/openstack-common/+spec/wsgi-common Ghe Rivero Is there a code metrics site that continually reports on metrics like duplication? Adding Ghe's report to a metric site would be the first step. That has always been a starting point as it gives code reviewers quick evaluation criteria to stop duplication before it ends up in trunk. Going at it directly fixes it looking backward but the duplication ends up back int the code eventually. The reports help fix the issue going forward.___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] wsgi code duplication
Hi Ghe, I suppose it will be very useful. We are planning to create a new project for nova-volumes (cider?) and I’m sure it will have same duplicate classes. Sooner we will have a common openstack layer is better. Regards, -Vladimir *From:* openstack-bounces+vladimir=zadarastorage@lists.launchpad.net[mailto: openstack-bounces+vladimir=zadarastorage@lists.launchpad.net] *On Behalf Of *Ghe Rivero *Sent:* Tuesday, April 24, 2012 7:28 AM *To:* Mark McLoughlin *Cc:* openstack *Subject:* Re: [Openstack] wsgi code duplication On Tue, Apr 24, 2012 at 2:40 PM, Mark McLoughlin mar...@redhat.com wrote: Hi Ghe, On Tue, 2012-04-24 at 12:15 +0200, Ghe Rivero wrote: Hi Everyone, i've been looking through wsgi code, and i have found a lot of duplicated code between all the projects. Thanks for looking into this. It sounds quite daunting. I wonder could we do this iteratively by extract the code which is most common into openstack-common, move the projects over to that and then start again on the next layer? Cheers, Mark. I have plans to try to move as much as possible into openstack-common. I will start with nova as a test bed and see what we get from there. My future plans include db code and tests (in the case of quantum, plugins test also have a lot of duplicated code). I register a bp for the wsgi issue: https://blueprints.launchpad.net/openstack-common/+spec/wsgi-common Ghe Rivero -- Ghe Rivero *OpenStack Distribution Engineer www.stackops.com | * ghe.riv...@stackops.com diego.parri...@stackops.com | +34 625 63 45 23 | skype:ghe.rivero* * http://www.stackops.com/ ADVERTENCIA LEGAL Le informamos, como destinatario de este mensaje, que el correo electrónico y las comunicaciones por medio de Internet no permiten asegurar ni garantizar la confidencialidad de los mensajes transmitidos, así como tampoco su integridad o su correcta recepción, por lo que STACKOPS TECHNOLOGIES S.L. no asume responsabilidad alguna por tales circunstancias. Si no consintiese en la utilización del correo electrónico o de las comunicaciones vía Internet le rogamos nos lo comunique y ponga en nuestro conocimiento de manera inmediata. Este mensaje va dirigido, de manera exclusiva, a su destinatario y contiene información confidencial y sujeta al secreto profesional, cuya divulgación no está permitida por la ley. En caso de haber recibido este mensaje por error, le rogamos que, de forma inmediata, nos lo comunique mediante correo electrónico remitido a nuestra atención y proceda a su eliminación, así como a la de cualquier documento adjunto al mismo. Asimismo, le comunicamos que la distribución, copia o utilización de este mensaje, o de cualquier documento adjunto al mismo, cualquiera que fuera su finalidad, están prohibidas por la ley. * PRIVILEGED AND CONFIDENTIAL We hereby inform you, as addressee of this message, that e-mail and Internet do not guarantee the confidentiality, nor the completeness or proper reception of the messages sent and, thus, STACKOPS TECHNOLOGIES S.L. does not assume any liability for those circumstances. Should you not agree to the use of e-mail or to communications via Internet, you are kindly requested to notify us immediately. This message is intended exclusively for the person to whom it is addressed and contains privileged and confidential information protected from disclosure by law. If you are not the addressee indicated in this message, you should immediately delete it and any attachments and notify the sender by reply e-mail. In such case, you are hereby notified that any dissemination, distribution, copying or use of this message or any attachments, for any purpose, is strictly prohibited by law. ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] wsgi code duplication
On Apr 24, 2012, at 10:55 AM, Thompson Lee wrote: On Apr 24, 2012, at 9:28 AM, Ghe Rivero wrote: I have plans to try to move as much as possible into openstack-common. I will start with nova as a test bed and see what we get from there. My future plans include db code and tests (in the case of quantum, plugins test also have a lot of duplicated code). I register a bp for the wsgi issue: https://blueprints.launchpad.net/openstack-common/+spec/wsgi-common Ghe Rivero Is there a code metrics site that continually reports on metrics like duplication? Adding Ghe's report to a metric site would be the first step. That has always been a starting point as it gives code reviewers quick evaluation criteria to stop duplication before it ends up in trunk. Going at it directly fixes it looking backward but the duplication ends up back int the code eventually. The reports help fix the issue going forward. I don't know of any duplication metrics being calculated, but Jenkins continually reports test coverage metrics: https://jenkins.openstack.org/portlet/dashboard_portlet_30/ Take care, Lorin -- Lorin Hochstein Lead Architect - Cloud Services Nimbis Services, Inc. www.nimbisservices.com smime.p7s Description: S/MIME cryptographic signature ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
