Re: [foreman-dev] Speed up rake apipie:cache:index
On Mon, Mar 6, 2017 at 10:11 AM, Martin Bačovský wrote: > The current state is that the apipie cache consists of root index page and > page per API resource and method and JSON desription of whole API > (multiplied by number of languages). Currently we build all the resource > specific pages during the package build time. The index and JSON need to > contain all the resources according to current configuration including all > the plugins that extends API. That is the reason we have to build this > during installation. > > During my benchmarks the bottleneck was loading the whole rails env. Before > the rake started to generate anything it took almost a minute on average > machine. May be different now though. > > Where we can surely spare some time is rpm package installation. Every > plugin extending API calls apipie:rake:index in %postrans resulting in about > 3 consequent redundant calls for default Katello install. If we can add some > state info that index was already built in this RPM transaction we could > spare a few mins. The state check could be useful. > > I'm not sure if JS would help us with creation of the JSON which is the most > used part of the API docs (used for Hammer and bindings), but I can imagine > it could be put together dynamically on first request from pieces generated > during a build time and cached. This way we could avoid index creation > completely. Do you have any benchmark how long it takes to generate json only vs json + html? Since some plugins currently modify foreman api endpoints and can potentially change (or remove) the api params too I don't think there's easy way of generating docs json for each plugin and merging it later. > > Martin > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
Guys if there's any doable way of getting rid of index generation or making it significantly faster, that would be super useful. LZ On Mon, Mar 6, 2017 at 10:11 AM, Martin Bačovský wrote: > The current state is that the apipie cache consists of root index page and > page per API resource and method and JSON desription of whole API > (multiplied by number of languages). Currently we build all the resource > specific pages during the package build time. The index and JSON need to > contain all the resources according to current configuration including all > the plugins that extends API. That is the reason we have to build this > during installation. > > During my benchmarks the bottleneck was loading the whole rails env. Before > the rake started to generate anything it took almost a minute on average > machine. May be different now though. > > Where we can surely spare some time is rpm package installation. Every > plugin extending API calls apipie:rake:index in %postrans resulting in about > 3 consequent redundant calls for default Katello install. If we can add some > state info that index was already built in this RPM transaction we could > spare a few mins. > > I'm not sure if JS would help us with creation of the JSON which is the most > used part of the API docs (used for Hammer and bindings), but I can imagine > it could be put together dynamically on first request from pieces generated > during a build time and cached. This way we could avoid index creation > completely. > > Martin > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
On Mar 6, 2017 11:11 AM, "Martin Bačovský" wrote: The current state is that the apipie cache consists of root index page and page per API resource and method and JSON desription of whole API (multiplied by number of languages). Currently we build all the resource specific pages during the package build time. The index and JSON need to contain all the resources according to current configuration including all the plugins that extends API. That is the reason we have to build this during installation. During my benchmarks the bottleneck was loading the whole rails env. Before the rake started to generate anything it took almost a minute on average machine. May be different now though. Where we can surely spare some time is rpm package installation. Every plugin extending API calls apipie:rake:index in %postrans resulting in about 3 consequent redundant calls for default Katello install. If we can add some state info that index was already built in this RPM transaction we could spare a few mins. I'm not sure if JS would help us with creation of the JSON which is the most used part of the API docs (used for Hammer and bindings), but I can imagine it could be put together dynamically on first request from pieces generated during a build time and cached. This way we could avoid index creation completely. Can we delegate this to an active job post startup? This will align with other efforts currently? Ohad Martin -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
The current state is that the apipie cache consists of root index page and page per API resource and method and JSON desription of whole API (multiplied by number of languages). Currently we build all the resource specific pages during the package build time. The index and JSON need to contain all the resources according to current configuration including all the plugins that extends API. That is the reason we have to build this during installation. During my benchmarks the bottleneck was loading the whole rails env. Before the rake started to generate anything it took almost a minute on average machine. May be different now though. Where we can surely spare some time is rpm package installation. Every plugin extending API calls apipie:rake:index in %postrans resulting in about 3 consequent redundant calls for default Katello install. If we can add some state info that index was already built in this RPM transaction we could spare a few mins. I'm not sure if JS would help us with creation of the JSON which is the most used part of the API docs (used for Hammer and bindings), but I can imagine it could be put together dynamically on first request from pieces generated during a build time and cached. This way we could avoid index creation completely. Martin -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
On Mon, Mar 06, 2017 at 10:46:18AM +0200, Ohad Levy wrote: > On Mon, Mar 6, 2017 at 10:34 AM, Tomas Strachota > wrote: > > > It's used for generating both html doc pages and json (consumed by > > apipie bindings). We can't get rid of iterating over languages, but > > generating docs only in json format and having html page that loads it > > async and generates the content dynamically would probably bring the > > desired speedup. I like it. > > > > for my own setup, I've always used: > FOREMAN_APIPIE_LANGS=en rake apipie:cache:index > > maybe we should consider creating a per language meta package (to include > the translations + api cache)? Since the puppet-foreman (and thus foreman-installer) also generates the index, it would make it difficult. IIRC the cache is regenated there because plugins might hook into it so that's something to consider. I do like the idea of just generating the JSON and having static HTML + JS pages that consume the JSON. > Ohad > > > > > T. > > > > > > On Mon, Mar 6, 2017 at 9:06 AM, Lukas Zapletal wrote: > > > Perhaps, but AFAIK we generate index page in all supported i18ned > > > languages, so the process is somehow repeated 12 times which makes is > > > slow I believe. > > > > > > LZ > > > > > > On Fri, Mar 3, 2017 at 5:11 PM, Ewoud Kohl van Wijngaarden > > > wrote: > > >> On Fri, Mar 03, 2017 at 04:34:49PM +0100, Lukas Zapletal wrote: > > >>> the rake apipie:cache:index command which we do both in build phase > > >>> and after RPM package installation is slow as hell. What is the reason > > >>> for that? Can't we redesign this to completely avoid need of > > >>> generating this kind of index? For example the page could be > > >>> JavaScript based, plugins would simply drop some metadata files and JS > > >>> would parse them hiding/showing appropriate sections on the index > > >>> page. This should not be too hard, but it will increase user > > >>> experience with installing and updating which we need to improve for > > >>> sure. > > >> > > >> Isn't the cache also used by apipie client bindings (which are used by > > >> tools like hammer)? > > >> > > >> -- > > >> You received this message because you are subscribed to the Google > > Groups "foreman-dev" group. > > >> To unsubscribe from this group and stop receiving emails from it, send > > an email to foreman-dev+unsubscr...@googlegroups.com. > > >> For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > -- > > > Later, > > > Lukas @lzap Zapletal > > > > > > -- > > > You received this message because you are subscribed to the Google > > Groups "foreman-dev" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > an email to foreman-dev+unsubscr...@googlegroups.com. > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to foreman-dev+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
On Mon, Mar 6, 2017 at 10:34 AM, Tomas Strachota wrote: > It's used for generating both html doc pages and json (consumed by > apipie bindings). We can't get rid of iterating over languages, but > generating docs only in json format and having html page that loads it > async and generates the content dynamically would probably bring the > desired speedup. I like it. > for my own setup, I've always used: FOREMAN_APIPIE_LANGS=en rake apipie:cache:index maybe we should consider creating a per language meta package (to include the translations + api cache)? Ohad > > T. > > > On Mon, Mar 6, 2017 at 9:06 AM, Lukas Zapletal wrote: > > Perhaps, but AFAIK we generate index page in all supported i18ned > > languages, so the process is somehow repeated 12 times which makes is > > slow I believe. > > > > LZ > > > > On Fri, Mar 3, 2017 at 5:11 PM, Ewoud Kohl van Wijngaarden > > wrote: > >> On Fri, Mar 03, 2017 at 04:34:49PM +0100, Lukas Zapletal wrote: > >>> the rake apipie:cache:index command which we do both in build phase > >>> and after RPM package installation is slow as hell. What is the reason > >>> for that? Can't we redesign this to completely avoid need of > >>> generating this kind of index? For example the page could be > >>> JavaScript based, plugins would simply drop some metadata files and JS > >>> would parse them hiding/showing appropriate sections on the index > >>> page. This should not be too hard, but it will increase user > >>> experience with installing and updating which we need to improve for > >>> sure. > >> > >> Isn't the cache also used by apipie client bindings (which are used by > >> tools like hammer)? > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev+unsubscr...@googlegroups.com. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > > > -- > > Later, > > Lukas @lzap Zapletal > > > > -- > > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
It's used for generating both html doc pages and json (consumed by apipie bindings). We can't get rid of iterating over languages, but generating docs only in json format and having html page that loads it async and generates the content dynamically would probably bring the desired speedup. I like it. T. On Mon, Mar 6, 2017 at 9:06 AM, Lukas Zapletal wrote: > Perhaps, but AFAIK we generate index page in all supported i18ned > languages, so the process is somehow repeated 12 times which makes is > slow I believe. > > LZ > > On Fri, Mar 3, 2017 at 5:11 PM, Ewoud Kohl van Wijngaarden > wrote: >> On Fri, Mar 03, 2017 at 04:34:49PM +0100, Lukas Zapletal wrote: >>> the rake apipie:cache:index command which we do both in build phase >>> and after RPM package installation is slow as hell. What is the reason >>> for that? Can't we redesign this to completely avoid need of >>> generating this kind of index? For example the page could be >>> JavaScript based, plugins would simply drop some metadata files and JS >>> would parse them hiding/showing appropriate sections on the index >>> page. This should not be too hard, but it will increase user >>> experience with installing and updating which we need to improve for >>> sure. >> >> Isn't the cache also used by apipie client bindings (which are used by >> tools like hammer)? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > -- > Later, > Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
Perhaps, but AFAIK we generate index page in all supported i18ned languages, so the process is somehow repeated 12 times which makes is slow I believe. LZ On Fri, Mar 3, 2017 at 5:11 PM, Ewoud Kohl van Wijngaarden wrote: > On Fri, Mar 03, 2017 at 04:34:49PM +0100, Lukas Zapletal wrote: >> the rake apipie:cache:index command which we do both in build phase >> and after RPM package installation is slow as hell. What is the reason >> for that? Can't we redesign this to completely avoid need of >> generating this kind of index? For example the page could be >> JavaScript based, plugins would simply drop some metadata files and JS >> would parse them hiding/showing appropriate sections on the index >> page. This should not be too hard, but it will increase user >> experience with installing and updating which we need to improve for >> sure. > > Isn't the cache also used by apipie client bindings (which are used by > tools like hammer)? > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [foreman-dev] Speed up rake apipie:cache:index
On Fri, Mar 03, 2017 at 04:34:49PM +0100, Lukas Zapletal wrote: > the rake apipie:cache:index command which we do both in build phase > and after RPM package installation is slow as hell. What is the reason > for that? Can't we redesign this to completely avoid need of > generating this kind of index? For example the page could be > JavaScript based, plugins would simply drop some metadata files and JS > would parse them hiding/showing appropriate sections on the index > page. This should not be too hard, but it will increase user > experience with installing and updating which we need to improve for > sure. Isn't the cache also used by apipie client bindings (which are used by tools like hammer)? -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[foreman-dev] Speed up rake apipie:cache:index
Hello, the rake apipie:cache:index command which we do both in build phase and after RPM package installation is slow as hell. What is the reason for that? Can't we redesign this to completely avoid need of generating this kind of index? For example the page could be JavaScript based, plugins would simply drop some metadata files and JS would parse them hiding/showing appropriate sections on the index page. This should not be too hard, but it will increase user experience with installing and updating which we need to improve for sure. -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
