Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 18.8.2016 11:06, David Kupka wrote: On 17/08/16 14:17, Jan Cholasta wrote: On 17.8.2016 13:21, David Kupka wrote: On 08/08/16 13:26, Jan Cholasta wrote: On 4.8.2016 16:32, David Kupka wrote: On 03/08/16 16:33, Jan Cholasta wrote: On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 I've made and reviewer overlooked some errors. Attached patches fix them. Shame on me. Anyway, 1) When schema() raises SchemaUpToDate, the current code explodes: ipa: ERROR: KeyError: 'fingerprint' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585, in load_plugins for package in self.packages: File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", line 92, in get_package plugins = schema.get_package(api, server_info, client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 558, in get_package fingerprint = str(schema['fingerprint']) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 483, in __getitem__ return self._dict[key] KeyError: 'fingerprint' 2) We read server info every time get_package() is called. It should be cache similarly to how the schema is cached in the api object. 3) Some of the commands are still fully initialized during "ipa help commands". Honza Updated patches attached. Thanks, ACK. Pushed to master: 6e6cbda036559e741ead0ab5ba18b0be0b41621e When locale is not available setlocale() explodes. Handling this exception in the same way as in ipalib/rpc.py to behave consistently. Works for me, ACK. Pushed to master: b6d5ed139b261b5db078ab652d22ea1d3b8092d3 -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 17/08/16 14:17, Jan Cholasta wrote: On 17.8.2016 13:21, David Kupka wrote: On 08/08/16 13:26, Jan Cholasta wrote: On 4.8.2016 16:32, David Kupka wrote: On 03/08/16 16:33, Jan Cholasta wrote: On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 I've made and reviewer overlooked some errors. Attached patches fix them. Shame on me. Anyway, 1) When schema() raises SchemaUpToDate, the current code explodes: ipa: ERROR: KeyError: 'fingerprint' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585, in load_plugins for package in self.packages: File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", line 92, in get_package plugins = schema.get_package(api, server_info, client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 558, in get_package fingerprint = str(schema['fingerprint']) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 483, in __getitem__ return self._dict[key] KeyError: 'fingerprint' 2) We read server info every time get_package() is called. It should be cache similarly to how the schema is cached in the api object. 3) Some of the commands are still fully initialized during "ipa help commands". Honza Updated patches attached. Thanks, ACK. Pushed to master: 6e6cbda036559e741ead0ab5ba18b0be0b41621e When locale is not available setlocale() explodes. Handling this exception in the same way as in ipalib/rpc.py to behave consistently. -- David Kupka From 083ad6016999eb244dc9915ec5fe6988f9053b0d Mon Sep 17 00:00:00 2001 From: David Kupka Date: Thu, 18 Aug 2016 10:59:09 +0200 Subject: [PATCH] schema cache: Fallback to 'en_us' when locale is not available https://fedorahosted.org/freeipa/ticket/6204 --- ipaclient/remote_plugins/schema.py | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 17.8.2016 13:21, David Kupka wrote: On 08/08/16 13:26, Jan Cholasta wrote: On 4.8.2016 16:32, David Kupka wrote: On 03/08/16 16:33, Jan Cholasta wrote: On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 I've made and reviewer overlooked some errors. Attached patches fix them. Shame on me. Anyway, 1) When schema() raises SchemaUpToDate, the current code explodes: ipa: ERROR: KeyError: 'fingerprint' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585, in load_plugins for package in self.packages: File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", line 92, in get_package plugins = schema.get_package(api, server_info, client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 558, in get_package fingerprint = str(schema['fingerprint']) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 483, in __getitem__ return self._dict[key] KeyError: 'fingerprint' 2) We read server info every time get_package() is called. It should be cache similarly to how the schema is cached in the api object. 3) Some of the commands are still fully initialized during "ipa help commands". Honza Updated patches attached. Thanks, ACK. Pushed to master: 6e6cbda036559e741ead0ab5ba18b0be0b41621e -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 08/08/16 13:26, Jan Cholasta wrote: On 4.8.2016 16:32, David Kupka wrote: On 03/08/16 16:33, Jan Cholasta wrote: On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 I've made and reviewer overlooked some errors. Attached patches fix them. Shame on me. Anyway, 1) When schema() raises SchemaUpToDate, the current code explodes: ipa: ERROR: KeyError: 'fingerprint' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585, in load_plugins for package in self.packages: File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", line 92, in get_package plugins = schema.get_package(api, server_info, client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 558, in get_package fingerprint = str(schema['fingerprint']) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 483, in __getitem__ return self._dict[key] KeyError: 'fingerprint' 2) We read server info every time get_package() is called. It should be cache similarly to how the schema is cached in the api object. 3) Some of the commands are still fully initialized during "ipa help commands". Honza Updated patches attached. -- David Kupka From 48c908746459f31b44a09045d53653ab503698ad Mon Sep 17 00:00:00 2001 From: David Kupka Date: Thu, 4 Aug 2016 16:02:24 +0200 Subject: [PATCH 01/10] schema cache: Do not reset ServerInfo dirty flag Once dirty flag is set to True it must not be set back to False. Otherwise changes are not written back to file. https://fedorahosted.org/freeipa/ticket/6048 --- ipaclient/remote_plugins/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ipaclient/remote_plugins/__init__.py b/ipaclient/remote_plugins/__init__.py index 444651d30fd0cd96299fecb7ee7b5e4532b0abd4..976d6968724088d8e1fe8d3615990accf585ffeb 100644 --- a/ipaclient/remote_pl
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 4.8.2016 16:32, David Kupka wrote: On 03/08/16 16:33, Jan Cholasta wrote: On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 I've made and reviewer overlooked some errors. Attached patches fix them. Shame on me. Anyway, 1) When schema() raises SchemaUpToDate, the current code explodes: ipa: ERROR: KeyError: 'fingerprint' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1348, in run api.finalize() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 707, in finalize self.__do_if_not_done('load_plugins') File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 422, in __do_if_not_done getattr(self, name)() File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 585, in load_plugins for package in self.packages: File "/usr/lib/python2.7/site-packages/ipalib/__init__.py", line 919, in packages ipaclient.remote_plugins.get_package(self), File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/__init__.py", line 92, in get_package plugins = schema.get_package(api, server_info, client) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 558, in get_package fingerprint = str(schema['fingerprint']) File "/usr/lib/python2.7/site-packages/ipaclient/remote_plugins/schema.py", line 483, in __getitem__ return self._dict[key] KeyError: 'fingerprint' 2) We read server info every time get_package() is called. It should be cache similarly to how the schema is cached in the api object. 3) Some of the commands are still fully initialized during "ipa help commands". Honza -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 03/08/16 16:33, Jan Cholasta wrote: On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 I've made and reviewer overlooked some errors. Attached patches fix them. -- David Kupka From ae1a9d024e37a153b6e9e4ada657f0e1e78300ef Mon Sep 17 00:00:00 2001 From: David Kupka Date: Thu, 4 Aug 2016 16:02:24 +0200 Subject: [PATCH 1/3] schema cache: Do not reset ServerInfo dirty flag Once dirty flag is set to True it must not be set back to False. Otherwise changes are not written back to file. https://fedorahosted.org/freeipa/ticket/6048 --- ipaclient/remote_plugins/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/ipaclient/remote_plugins/__init__.py b/ipaclient/remote_plugins/__init__.py index 444651d30fd0cd96299fecb7ee7b5e4532b0abd4..976d6968724088d8e1fe8d3615990accf585ffeb 100644 --- a/ipaclient/remote_plugins/__init__.py +++ b/ipaclient/remote_plugins/__init__.py @@ -59,7 +59,8 @@ class ServerInfo(collections.MutableMapping): return self._dict[key] def __setitem__(self, key, value): -self._dirty = key not in self._dict or self._dict[key] != value +if key not in self._dict or self._dict[key] != value: +self._dirty = True self._dict[key] = value def __delitem__(self, key): -- 2.7.4 From a6e9b17e0ad109f3237f4417828f7a3cfb5aa743 Mon Sep 17 00:00:00 2001 From: David Kupka Date: Thu, 4 Aug 2016 16:07:08 +0200 Subject: [PATCH 2/3] schema cache: Write format information in cache When format is not in cache '0' is assumed and client expecting higher format refuses to load such cache. https://fedorahosted.org/freeipa/ticket/6048 --- ipaclient/remote_plugins/schema.py | 1 + 1 file changed, 1 insertion(+) diff --git a/ipaclient/remote_plugins/schema.py b/ipaclient/remote_plugins/schema.py index a215452ea0d2c1278a6121b3806b0daee02abd6e..13b1c365b111faae673c5ed78a7e5439a869c330 100644 --- a/ipaclient/remote_plugins/schema.py +++ b/ipaclient/remote_plugins/schema.py @@ -524,6 +524,7 @@ class Schema(object): schema.writestr('_help', json.dumps(self._generate_help(self._dict))) +schema.writestr('format', json.dumps(FORMAT)) def _read(self, path): with self._open_schema(self._fingerprint, 'r') as zf: -- 2.7.4 From c57a1b98ca518db8909d4d1176c99c4b2daec606 Mon Sep 17 00:00:00 2001 From: David Kupka Date: Thu,
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 3.8.2016 16:23, David Kupka wrote: On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. Thanks, ACK. Pushed to master: 229e2a1ed9ea9877cb5e879fadd99f9040f77c96 -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 21/07/16 10:12, Jan Cholasta wrote: Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza Updated patches attached. -- David Kupka From bae93c2a57b51c8b5c11d481f9df2a62f330fb81 Mon Sep 17 00:00:00 2001 From: David Kupka Date: Wed, 27 Jul 2016 10:46:40 +0200 Subject: [PATCH 1/6] schema: Speed up schema cache Check presence of schema in cache (and download it if necessary) on __init__ instead of with each __getitem__ call. Prefill internal dictionary with empty record for each command to be able to quickly determine if requested command exist in schema or not. Rest of schema data are read from cache on first attempt to retrive them. https://fedorahosted.org/freeipa/ticket/6048 https://fedorahosted.org/freeipa/ticket/6069 --- ipaclient/remote_plugins/schema.py | 297 ++--- 1 file changed, 175 insertions(+), 122 deletions(-) diff --git a/ipaclient/remote_plugins/schema.py b/ipaclient/remote_plugins/schema.py index cd1d5d607978899254325f634ccec91d2c92f59b..70cd7536d836ec113236bfdae8bbabb2d843725d 100644 --- a/ipaclient/remote_plugins/schema.py +++ b/ipaclient/remote_plugins/schema.py @@ -5,10 +5,8 @@ import collections import errno import fcntl -import glob import json import os -import re import sys import time import types @@ -65,8 +63,6 @@ USER_CACHE_PATH = ( '.cache' ) ) -SCHEMA_DIR = os.path.join(USER_CACHE_PATH, 'ipa', 'schema') -SERVERS_DIR = os.path.join(USER_CACHE_PATH, 'ipa', 'servers') logger = log_mgr.get_logger(__name__) @@ -274,15 +270,6 @@ class _SchemaObjectPlugin(_SchemaPlugin): schema_key = 'classes' -def _ensure_dir_created(d): -try: -os.makedirs(d) -except OSError as e: -if e.errno != errno.EEXIST: -raise RuntimeError("Unable to create cache directory: {}" - "".format(e)) - - class _LockedZipFile(zipfile.ZipFile): """ Add locking to zipfile.ZipFile Shared lock is used with read mode, exclusive with write mode. @@ -308,7 +295,10 @@ class _SchemaNameSpace(collections.Mapping): self._schema = schema def __getitem__(self, key): -return self._schema.read_namespace_member(self.name, key) +try: +return self._schema.read_namespace_member(self.name, key) +except KeyError: +raise KeyError(key) def __iter__(self): for key in self._schema.iter_namespace(self.name): @@ -322,6 +312,62 @@ class NotAvailable(Exception): pass +class ServerInfo(collections.MutableMapping): +_DIR = os.path.join(USER_CACH
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
Hi, On 20.7.2016 14:32, David Kupka wrote: On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. Patch 0112: I think this would better be done in Schema.read_namespace_member, because Schema is where all the state is. (BTW does _SchemaNameSpace.__getitem__ raise KeyError for non-existent keys? It looks like it doesn't.) Patch 0113: How about setting _schema_cached to False in Schema.__init__() rather that getattr()-ing it in _ensure_cached()? Patch 0116: ClientCommand.doc should be a class property as well, otherwise .summary won't work on it correctly. _SchemaCommand.doc should not be a property, as it's not needed for .summary to work on it correctly. Otherwise works fine for me. Honza -- Jan Cholasta -- Manage your subscription for the Freeipa-devel mailing list: https://www.redhat.com/mailman/listinfo/freeipa-devel Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code
Re: [Freeipa-devel] [PATCH 0112-7] Speeding up cli help
On 15/07/16 12:53, David Kupka wrote: Hello! After Honza introduced thin client that builds plugins and commands dynamically from schema client became much slower. This is only logical, instead of importing a module client now must fetch the schema from server, parse it and instantiate the commands using the data. First step to speed it up was addition of schema cache to client. That removed the RTT and download time of fetching schema every time. Now the most time consuming task became displaying help for lists of topics and command and displaying individual topics. This is simply because of the need to instantiate all the commands to find the relations between topics and commands. All the necessary bits for server commands and topics are already in the schema cache so we can skip this part and generate help from it, right? Not so fast! There are client plugins with commands and topics. So we can generate basic bits (list of all topics, list of all commands, list of commands for each topic) from schema and store it in cache. Then we need to go through all client plugins and get similar bits for client plugins. Then we can merge and print. Still the client response is not as fast as before and I this it even can't be. Also first time you display particular topic or list takes longer because it must be freshly generated and stored in cache for next use. And this is what the attached patches do. https://fedorahosted.org/freeipa/ticket/6048 Reimplemented so there is no need to distinguish client plugins and remote plugins. The main idea of this approach is to avoid creating instances of the commands just to get the information about topic, name and summary needed for displaying help. Instead class properties are used to access the information directly in schema. -- David Kupka From 8eefe955f7e961e1a4e55b823772383243c029a5 Mon Sep 17 00:00:00 2001 From: David Kupka Date: Tue, 19 Jul 2016 09:34:35 +0200 Subject: [PATCH 1/6] schema: Read data from cache only once Keep data when read from storage to avoid unnecessary reads. https://fedorahosted.org/freeipa/ticket/6048 --- ipaclient/remote_plugins/schema.py | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/ipaclient/remote_plugins/schema.py b/ipaclient/remote_plugins/schema.py index cd1d5d607978899254325f634ccec91d2c92f59b..09f041592f04e65544cf65647f28f395a2f9e4a7 100644 --- a/ipaclient/remote_plugins/schema.py +++ b/ipaclient/remote_plugins/schema.py @@ -306,9 +306,17 @@ class _SchemaNameSpace(collections.Mapping): def __init__(self, schema, name): self.name = name self._schema = schema +self._dict = {} def __getitem__(self, key): -return self._schema.read_namespace_member(self.name, key) +try: +return self._dict[key] +except KeyError: +self._dict[key] = self._schema.read_namespace_member( +self.name, +key +) +return self._dict[key] def __iter__(self): for key in self._schema.iter_namespace(self.name): -- 2.7.4 From fe155db233456b2d6d917a010177eb382a3de58f Mon Sep 17 00:00:00 2001 From: David Kupka Date: Wed, 20 Jul 2016 09:54:45 +0200 Subject: [PATCH 2/6] schema: Speed up cache verification Check schema only once for each api instance. https://fedorahosted.org/freeipa/ticket/6048 --- ipaclient/remote_plugins/schema.py | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/ipaclient/remote_plugins/schema.py b/ipaclient/remote_plugins/schema.py index 09f041592f04e65544cf65647f28f395a2f9e4a7..92430950f96be266ebaa2f238de226e1c02b8e6c 100644 --- a/ipaclient/remote_plugins/schema.py +++ b/ipaclient/remote_plugins/schema.py @@ -403,6 +403,9 @@ class Schema(object): return (fp, exp) def _ensure_cached(self): +if getattr(self, '_schema_cached', False): +return + no_info = False try: # pylint: disable=access-member-before-definition @@ -422,13 +425,9 @@ class Schema(object): logger.warning('Failed to load server properties: {}' ''.format(e)) -force_check = ((not getattr(self, '_schema_checked', False)) and - self._api.env.force_schema_check) - -if (force_check or +if (self._api.env.force_schema_check or no_info or exp < time.time() or not Schema._in_cache(fp)): (fp, exp) = self._get_schema() -self._schema_checked = True _ensure_dir_created(SERVERS_DIR) try: with self._open_server_info(self._api.env.server, 'w') as sc: @@ -436,6 +435,7 @@ class Schema(object): except Exception as e: logger.warning('Failed to store server properties: {}' ''.format(e)) +self._schema_cached = True if not self._dict: