Re: [foreman-dev] Proposal: Foreman 1.18 = 2.0

[sshtein]

About extracting puppet to a plugin:

Preparation work for it is tracked here. 

It has been a while from my latest check, but I don't think we are too far 
from getting it done. I think 2 releases should be enough for changing 
foreman's code, depending on the amount of attention new PR's will get.
There is also a big change in apipie that needs to get in, if we want 
proper properties description in our API.

One thing that we did not account for is the amount of work in plugins that 
depend on puppet - both Discovery and Remote Execution have references to 
puppet.


On Thursday, November 30, 2017 at 2:26:24 PM UTC+2, Marek Hulán wrote:
>
> Dne čtvrtek 30. listopadu 2017 11:59:27 CET, Ewoud Kohl van Wijngaarden 
> napsal(a): 
> > On Thu, Nov 30, 2017 at 09:57:24AM +0100, Marek Hulán wrote: 
> > >Dne středa 29. listopadu 2017 14:36:18 CET, Ewoud Kohl van Wijngaarden 
> > > 
> > >napsal(a): 
> > >> On Wed, Nov 29, 2017 at 02:18:35PM +0100, Lukas Zapletal wrote: 
> > >> >> Bikeshedding about SemVer aside, I'm good with doing a 2.0 release 
> in 
> > >> >> the near future, but *please* lets use it to deprecate / drop 
> stuff we 
> > >> >> no longer want to maintain. Otherwise there's no real point to it. 
> > >> > 
> > >> >I agree we can take this "opportunity" to drop some deprecated 
> things 
> > >> >like V1 API, but I don't see many other things. We are pretty good 
> in 
> > >> >deprecating things using our "two releases" rule which should be 
> > >> >followed no matter if we bump major version or not. 
> > >> 
> > >> +1 this is very similar to Django's release policy: in 1.x it was x 
> > >> deprecated and x+2 removed. Starting 2.0 they'll follow semver. I'd 
> > >> suggest we do the same. 
> > >> 
> > >> >Let's not block 2.0 with any feature, I wrote the reasons, if we fit 
> > >> >in some deprecation work why not. But's let's agree on 2.0 timeframe 
> > >> >regardless of any planning. 
> > >> 
> > >> +1 on letting 2.0 drop block on dropping things rather than adding 
> > >> things. 
> > > 
> > >Actually I'd prefer to use this opportunity to drop or rewrite stuff we 
> > >know is problematic. E.g. taxonomies (especially nesting), API v1, 
> > >hostgroup provisioning, extracting puppet to a plugin, smart variables 
> > >merging with parameters (not smart class parameters), dropping 
> unattended 
> > >installation mode (or at least refactoring). 
> > 
> > These sound like good goals. How hard would it be to make the changes 
> > you're suggesting? 
> > 
> > I have a slight preference to make 1.17 the 2.0 already because of the 
> > changes it's introducing but other than dropping API v1 and unattended 
> > installation mode they all sound like non-trivial tasks that will not be 
> > ready in time for 1.17. 
>
> very rough estimations 
> taxonomies (at least two release cycles) 
> apiv1 (easy if we don't need to spend time on proper extracting to 
> gem/plugin 
> that users could still install) 
> host group provisioning (drop is easy, refactor could be 1 foreman 
> release) 
> smart variables unification with params - 1 or 2 releases, Ori has done 
> some 
> research in this area 
> extracting puppet to a plugin, not sure, Shim would perhaps know better 
> unattended installation mode - 1 release 
>
> I think even if we paralellize the effort, they should all be delivered in 
> same release that we would call 2.0. Not sure how to do it, keeping PR 
> opened 
> for more than few weeks is not good :-) I'd like to avoid 2 develop 
> branches. 
>
> > Maybe we need a list of deprecated features/desired breaking changes so 
> > we can look at it after a release when we're starting to plan for the 
> > next. If it grows to a certain size or the release manager feels like 
> > it's time for a major then they can announce it's being considered. That 
> > way devs have time to give priority to the breaking changes a long time 
> > in advance rather than the 2 weeks prior to branching. 
>
> That sounds good to me. 
>
> > What I'm proposing are guidelines that mostly focus on clear 
> > communication - no strict policy. Communication is much more important 
> > than what we actually do. 
>
> +1 
>
> -- 
> Marek 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: [UX] Facets and Host UI - roadmap discussion.

[sshtein]

Marek, you lead me to an interesting conclusion:

I think we have to distinguish two things here - there are workflows (such 
as provisioning, config management, fact reporting) and there are 
information aspects.
An information aspect is a set of properties that describe a host in 
different system. For example puppet facet would store all properties that 
are needed to describe a host in puppet. Ovirt facet would store all 
properties that describe the host in ovirt system.
Workflow, on the other hand, is a set of actions that needed to be taken in 
a certain order to achieve some operational goal. Examples to that would be 
provisioning - a set of actions that involve different systems (dhcp, dns, 
vm infrastructure and OS installer) that result in a fully operational 
server. Another example would be monitoring - in this case we will want 
multiple systems (like puppet's facter, vm's power status e.t.c.) to report 
to the user.

Now, once we have those concepts, we can try and translate this into the 
UI: In my opinion, data entry should be done from screens centered on 
information aspects, regardless of the workflows where this information 
could be used. On the other hand each workflow deserves a "summary" read 
only screen, where we will combine data from multiple facets to show which 
data would be used for that particular workflow.

>From a more practical  point of view, our current screens serve both 
workflows and data entry. It means that we have to establish for each 
screen what it tries to achieve - either it's a data entry page, such as 
host's new\edit form or it's a workflow preview/result page such as host 
show page. Data entry pages should be rigidly grouped by facets, but 
workflow pages should be extensible, so each facet will be able to show 
relevant information on the same page.

How does this sound to you?
Roxanne, does it fit with your vision of form's next generation?



On Wednesday, November 15, 2017 at 8:33:35 AM UTC+2, Marek Hulan wrote:
>
> I think these screenshots illustrate that pure option 2 can have negative 
> impact on usability. If I'm a puppet user, the puppet environment is one 
> of 
> the most important thing to set. Having it in last tab completely separate 
> from hostgroup does not feel right. If some field of facet is part of 
> provisioning workflow, it should be extending provisioning UI/facet. 
> Another example from content management, the content source is definitely 
> a 
> part provisioning, I want to be able to choose content view as an 
> installation medium. 
>
> -- 
> Marek 
>
> -- 
> Marek 
>
>
>
> On November 12, 2017 19:36:14 ssh...@redhat.com  wrote: 
>
> > 
> > OK, I have managed to create some screenshots of the before and after 
> > state. Please don't judge the styling - it's more about the technical 
> > abilities than the styling. 
> > 
> > I will take Puppet as an example. Let's say we have puppet facet that 
> has 
> > the following data: puppet environment, puppet proxy and puppet ca proxy 
> > fields plus a list of puppet classes assigned to the host. 
> > 
> > Before: 
> > The information is spread on multiple screens: 
> > Take a look at this screenshot: https://ibb.co/bsXT8G it shows where 
> this 
> > information is currently located on the main tab. 
> > This screenshot: https://ibb.co/ksuaoG shows the detailed puppet 
> classes 
> > view. 
> > 
> > After: 
> > Everything related to puppet is presented on a single tab. This tab can 
> be 
> > enabled and disabled based on user's preference - the user can decide to 
> > turn puppet management on or off for this host. 
> > https://ibb.co/bNnd8G shows the tab when the fields are enabled, and 
> > https://ibb.co/eCJ72b shows all the fields disabled. 
> > 
> > I hope it helps to visualize both options. 
> > 
> > 
> > On Wednesday, November 8, 2017 at 12:08:06 AM UTC+2, Walden Raines 
> wrote: 
> >> 
> >> Hey Shim, 
> >> 
> >> Can you please include screenshots (or, even better, a quick video or 
> gif) 
> >> of the new UI to make it easier for people to visualize who don't have 
> the 
> >> code checked out? 
> >> 
> >> Assuming I'm understanding your description of the two options, I would 
> >> also vote for option #2 as option #1 sounds like it would be very 
> difficult 
> >> to ensure a good UI since some other plugin could just put whatever 
> they 
> >> want wherever in the UI. 
> >> 
> >> Thanks, 
> >> Walden 
> >> 
> >> 
> >> 
> >> On Tue, Nov 7, 2017 at 5:38 AM, Timo Goebel  >> > wrote: 
> >> 
> >>> I have been playing with Facets the last few weeks and must say, that 
> >>> they are really great. It's pretty easy to add dedicated functionality 
> to 
> >>> the host model and I want to use that for some of my plugins (Omaha, 
> >>> Monitoring, something new, ...). 
> >>> Everything is great so far except for the missing UI hooks Shim 
> mentions. 
> >>> 
> >>> What I want to do are mostly easy thing, like adding a new tab to the 
> >>> host form 

Re: [foreman-dev] Re: [UX] Facets and Host UI - roadmap discussion.

[sshtein]

OK, I have managed to create some screenshots of the before and after 
state. Please don't judge the styling - it's more about the technical 
abilities than the styling.

I will take Puppet as an example. Let's say we have puppet facet that has 
the following data: puppet environment, puppet proxy and puppet ca proxy 
fields plus a list of puppet classes assigned to the host.

Before: 
The information is spread on multiple screens:
Take a look at this screenshot: https://ibb.co/bsXT8G it shows where this 
information is currently located on the main tab.
This screenshot: https://ibb.co/ksuaoG shows the detailed puppet classes 
view.

After:
Everything related to puppet is presented on a single tab. This tab can be 
enabled and disabled based on user's preference - the user can decide to 
turn puppet management on or off for this host.
https://ibb.co/bNnd8G shows the tab when the fields are enabled, and 
https://ibb.co/eCJ72b shows all the fields disabled.

I hope it helps to visualize both options.


On Wednesday, November 8, 2017 at 12:08:06 AM UTC+2, Walden Raines wrote:
>
> Hey Shim,
>
> Can you please include screenshots (or, even better, a quick video or gif) 
> of the new UI to make it easier for people to visualize who don't have the 
> code checked out?
>
> Assuming I'm understanding your description of the two options, I would 
> also vote for option #2 as option #1 sounds like it would be very difficult 
> to ensure a good UI since some other plugin could just put whatever they 
> want wherever in the UI.  
>
> Thanks,
> Walden
>
>
>
> On Tue, Nov 7, 2017 at 5:38 AM, Timo Goebel  > wrote:
>
>> I have been playing with Facets the last few weeks and must say, that 
>> they are really great. It's pretty easy to add dedicated functionality to 
>> the host model and I want to use that for some of my plugins (Omaha, 
>> Monitoring, something new, ...).
>> Everything is great so far except for the missing UI hooks Shim mentions.
>>
>> What I want to do are mostly easy thing, like adding a new tab to the 
>> host form or host show page. Currently, the only way to do this is using 
>> deface.
>> But this feels pretty hacky to me and isn't good to maintain. I'd really 
>> appreciate if there were easy and tested hooks for common areas like the 
>> host show page.
>>
>> In my opinion, we are already too late on finishing the facets (and 
>> Pagelets) integration. Personally, I don't have a strong opinion on either 
>> option. But prefer the second approach as well.
>>
>> In regards to widening the feature gap for a host ux redesign: We have to 
>> provide extension points anyways. The Foreman community gains a lot of 
>> value from the rich plugin ecosystem and the possibility to extend Foreman 
>> fairly easy.
>> When we redo the host ux pages, we have to provide extension points. This 
>> is not a nice-to-have feature, but a must-have in my opinion.
>>
>> - Timo
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to foreman-dev...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: [POC] Automatic inspection of user-created provisioning templates

[sshtein]

Finally I have finished first stage of engine development - now there is a 
sinatra app that could be added to any rack app (although not tested) and 
can run as a standalone executable (for now using webrick).

Git repo is located here: https://github.com/ShimShtein/detective

Key files: 

   - lib/detective/web/endpoint.rb: Sinatra app that exposes the resources.
   - lib/detective/launcher.rb: Rubocop's CLI wrapper to make use of custom 
   cops.
   - bin/detective: Standalone mode executable.


I would really appreciate if you evaluate this code and see if it is 
something we want to work with.

One caveat though: I still didn't add the code responsible for erb 
handling, it will be added later.

Shim.



On Thursday, September 14, 2017 at 4:27:15 PM UTC+3, Shimon Shtein wrote:
>
> Ewoud:
>
> About rack endpoint: It should be created for use case 2, where the check 
> is running in "online"-ish mode. 
> I prefer creating it as a microservice for scalability reasons, so I 
> wouldn't want to tie it too tightly with foreman-templates. Besides that, 
> tying it into foreman-templates will mean the same life cycle that I would 
> prefer to avoid. 
> As I already wrote for case 1 and 3, we could use the underlying ruby 
> wrapper directly, hence avoid the usage of the API.
> As for additional dependencies, I probably would add passenger as an 
> optional (development) dependency, so you will have the full feature set on 
> development machines. In production it will check if if passenger is 
> available only if it runs in standalone mode. It will not ask for passenger 
> if someone has decided to use something else, like webrick (personally I 
> would recommend against webrick, it's not parallel at all which is 
> important here).
>
> About the command line tool, I have nothing against it, it should be 
> available in the gem.
>
>
>
>
>
> On Thu, Sep 14, 2017 at 2:06 PM, Ewoud Kohl van Wijngaarden <
> ew...@kohlvanwijngaarden.nl> wrote:
>
>> On Wed, Sep 13, 2017 at 12:05:49PM -0700, ssht...@redhat.com wrote:
>>
>>>
>>> First attempt to create a design. It's an open discussion, everyone who
>>> wants to chime in, please do.
>>>
>>> The engine: will be deployed as a separate gem. My name suggestion
>>> the-detective  
>>> (Sinatra
>>> plays a cop).
>>>
>>> It will wrap the invocation of rubocop with defaults and parameters 
>>> needed
>>> to support our use case:
>>> 1. Support for erb
>>> 2. Support for completely customized set of cops.
>>> 3. Parametrized list of folders containing cops to be added to the list.
>>>
>>> In addition it will add tooling to expose a rack endpoint for rubocop
>>> invocation:
>>> 1. List of all available cops (kind of metadata)
>>> 2. A POST method that receives a source file, list of cops, and output
>>> format that will return the result of rubocop's analysis.
>>> 3. Will be mountable to any Rails application
>>> 4. Will have an option to run as a standalone process (probably using
>>> passenger with sort-lived process retention settings, since its one 
>>> process
>>> per request nature)
>>>
>>
>> Why should it be a rack endpoint? My thinking was much more of a normal 
>> ruby API with a command line tool around it. There should be no passenger 
>> dependency to keep its dependencies small. foreman_templates can consume 
>> the ruby API and expose it to the user as it wants.
>>
>> Usage for foreman needs:
>>>
>>> Use case 1 (community templates CI):
>>> 1. Reference the detective gem from templates plugin.
>>> 2. Deploy foreman-core with templates plugin enabled.
>>> 3. Add rake task that will invoke rubocop on specified folder using
>>> detective's invocation wrapper.
>>>
>>
>> Ideally we'd have a light command line tool that does:
>>
>> detective \
>>  --cops /path/to/foreman/checkout/cops \
>>  --cops /path/to/katello/checkout/cops \
>>  --cops /path/to/other/plugin/with/cops \
>>  /path/to/some/template/dir \
>>  /path/to/another/template/dir
>>
>> That way we can do a simple git clone foreman in community-templates, 
>> bundle install and run it within Travis. This can indeed be wrapped in a 
>> rake task but given the paths can change on a developers workstation it is 
>> good to have an easy manual option.
>>
>> Use case 2 (Validate single template from templates UI)
>>> 1. Reference detective gem from templates plugin.
>>> 2. Add cops declaration ability to plugins in foreman core
>>> 3. Templates plugin is responsible for adding/maintaining detective's
>>> endpoint.
>>> 4. Foreman core exposes an option to add actions to template editing 
>>> screen.
>>> 5. Templates plugin uses extension point from 4 to add its own action 
>>> that
>>> will invoke detective's endpoint and modify template editor to show the
>>> result as linting (it's possible with ace and monaco).
>>>
>>> Use case 3 (upgrade scenario):
>>> As a first step, we can try and report broken templates after the 
>>> upgrade.
>>> It will be pretty 

[foreman-dev] Re: [POC] Automatic inspection of user-created provisioning templates

[sshtein]

First attempt to create a design. It's an open discussion, everyone who 
wants to chime in, please do.

The engine: will be deployed as a separate gem. My name suggestion 
the-detective  
(Sinatra 
plays a cop).

It will wrap the invocation of rubocop with defaults and parameters needed 
to support our use case:
1. Support for erb
2. Support for completely customized set of cops.
3. Parametrized list of folders containing cops to be added to the list.

In addition it will add tooling to expose a rack endpoint for rubocop 
invocation:
1. List of all available cops (kind of metadata)
2. A POST method that receives a source file, list of cops, and output 
format that will return the result of rubocop's analysis.
3. Will be mountable to any Rails application
4. Will have an option to run as a standalone process (probably using 
passenger with sort-lived process retention settings, since its one process 
per request nature)

Usage for foreman needs:

Use case 1 (community templates CI): 
1. Reference the detective gem from templates plugin.
2. Deploy foreman-core with templates plugin enabled.
3. Add rake task that will invoke rubocop on specified folder using 
detective's invocation wrapper.

Use case 2 (Validate single template from templates UI)
1. Reference detective gem from templates plugin.
2. Add cops declaration ability to plugins in foreman core
3. Templates plugin is responsible for adding/maintaining detective's 
endpoint.
4. Foreman core exposes an option to add actions to template editing screen.
5. Templates plugin uses extension point from 4 to add its own action that 
will invoke detective's endpoint and modify template editor to show the 
result as linting (it's possible with ace and monaco).

Use case 3 (upgrade scenario):
As a first step, we can try and report broken templates after the upgrade.
It will be pretty similar to community templates CI use case, only the 
templates code will be exported from user's database.


I want to start working on the engine gem as soon as possible, so I would 
really appreciate any inputs on the process before I have started with this 
implementation.

Shim.



On Wednesday, August 30, 2017 at 11:48:09 AM UTC+3, ssh...@redhat.com wrote:
>
>
> After a great talk on community demo 
> , here is a follow up with 
> the points that were raised during the discussion:
>
> Use cases:
>
>1. Run all cops as part of community templates CI against the whole 
>repository
>2. Run all cops against a single template invoked by the user from 
>template editing screen (foreman core)
>3. Upgrade scenario: Preferably run cops for the next foreman version 
>before the actual upgrade to make sure the templates will remain valid.
>
>
> Features:
>
>1. List of rues should be pluggable [Shim]: It looks like it is a 
>must-have for the engine.
>2. Deployment options
>1. Engine as a separate gem, cops in a relevant repository - core cops 
>   in core, plugin cops in plugins.
>   2. Engine with all cops in a single gem, versioned per foreman 
>   version.
>   3. Engine as part of templates plugin, cops as part of relevant 
>   plugins.
>   4. Separate gems for everything: foreman-cops-engine, 
>   foreman-cops-core, foreman-cops-plugin1, foreman-cops-plugin2 e.t.c. 
> Engine 
>   is versioned per foreman release version (for the sake of rubocop 
> version), 
>   cops are versioned per plugin version.
>
> General comments:
>
>1. Cops writing should be enforced on PR's that are changing the way 
>to write templates [mhulan]
>2. Cops are dependent on core/plugin version [gwmngilfen]
>
>
>
>
> On Monday, August 14, 2017 at 2:29:02 PM UTC+3, ssh...@redhat.com wrote:
>>
>> TL;DR: I have developed a way to scan any template and see if there are 
>> suspicious/incorrect code patterns in them, so the templates will remain 
>> valid even after foreman code changes.
>>
>> Recently I have started to think about user created templates and foreman 
>> upgrades.
>>
>> When user upgrades foreman, hist default templates get upgraded by the 
>> installer/migrations, but templates created by the user (both cloned and 
>> from scratch) are not touched.
>> This could lead to invalid templates and broken provisioning 
>> functionality for the user.
>> Good example for this would be the change 
>> 
>>  
>> from calling to <%= foreman_url %> to <%= foreman_url('built') %> 
>>
>> I was looking for a way to inspect any template, in order to identify 
>> problematic code as soon as the system is upgraded.
>>
>> I came down to a solution based on rubocop - it's already analyzing 
>> source files for patterns.
>> I have created a POC that analyzes a template written to a file, and 
>> presents the resulting errors as regular rubocop 

[foreman-dev] Re: [POC] Automatic inspection of user-created provisioning templates

[sshtein]

After a great talk on community demo 
, here is a follow up with the 
points that were raised during the discussion:

Use cases:

   1. Run all cops as part of community templates CI against the whole 
   repository
   2. Run all cops against a single template invoked by the user from 
   template editing screen (foreman core)
   3. Upgrade scenario: Preferably run cops for the next foreman version 
   before the actual upgrade to make sure the templates will remain valid.


Features:

   1. List of rues should be pluggable [Shim]: It looks like it is a 
   must-have for the engine.
   2. Deployment options
   1. Engine as a separate gem, cops in a relevant repository - core cops 
  in core, plugin cops in plugins.
  2. Engine with all cops in a single gem, versioned per foreman 
  version.
  3. Engine as part of templates plugin, cops as part of relevant 
  plugins.
  4. Separate gems for everything: foreman-cops-engine, 
  foreman-cops-core, foreman-cops-plugin1, foreman-cops-plugin2 e.t.c. 
Engine 
  is versioned per foreman release version (for the sake of rubocop 
version), 
  cops are versioned per plugin version.
   
General comments:

   1. Cops writing should be enforced on PR's that are changing the way to 
   write templates [mhulan]
   2. Cops are dependent on core/plugin version [gwmngilfen]




On Monday, August 14, 2017 at 2:29:02 PM UTC+3, ssh...@redhat.com wrote:
>
> TL;DR: I have developed a way to scan any template and see if there are 
> suspicious/incorrect code patterns in them, so the templates will remain 
> valid even after foreman code changes.
>
> Recently I have started to think about user created templates and foreman 
> upgrades.
>
> When user upgrades foreman, hist default templates get upgraded by the 
> installer/migrations, but templates created by the user (both cloned and 
> from scratch) are not touched.
> This could lead to invalid templates and broken provisioning functionality 
> for the user.
> Good example for this would be the change 
> 
>  
> from calling to <%= foreman_url %> to <%= foreman_url('built') %> 
>
> I was looking for a way to inspect any template, in order to identify 
> problematic code as soon as the system is upgraded.
>
> I came down to a solution based on rubocop - it's already analyzing source 
> files for patterns.
> I have created a POC that analyzes a template written to a file, and 
> presents the resulting errors as regular rubocop (clang style).
> All source codes are available as gist: 
> https://gist.github.com/ShimShtein/341b746f15826261053e97c2f435ff1a
>
> Small explanation for the gist:
>
> Entry point: inspect_template.rb
> Usage:
> Put everything from the gist to a single folder and execute:
>
>> inspect_template /path/to/template_source.erb
>
> This script aggregates all the parts that are needed to create the 
> clang-like output.
>
> The process:
>
>1. Strip all non-ruby text from the template. This is done by 
>erb_strip.rb. It turns everything that is not a ruby code into spaces, so 
>the ruby code remains in the same places as it was in the original file.
>2. Run rubocop with custom rules and erb monkey patch and produce a 
>json report
>   1. foreman_callback_cop.rb custom rule file. The most interesting 
>   line is "def_node_matcher :foreman_url_call?, '(send nil 
>   :foreman_url)'". Here you define which pattern to look for in the 
>   AST, in our case we are looking for calls (send) for foreman_url method 
>   without parameters.
>   2. foreman_erb_monkey_patch.rb file: Patches rubocop engine to 
>   treat *.erb files as source files and not skip them.
>3. Process the resulting json to convert it to clang style 
>highlighting.
>
> Possible usages:
>
>- Scanning all template after foreman upgrade to see that they are 
>still valid.
>- Linting a template while while editing.
>- Using rubocop's autocorrect feature to automatically fix offences 
>found by this process.
>
> Long shot: we can create custom rules to inspect code for our plugins too, 
> especially if we start creating custom rules in rubocop.
>
> I am interested in comments, opinions and usages to see how to proceed 
> from here.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] [POC] Automatic inspection of user-created provisioning templates

[sshtein]

TL;DR: I have developed a way to scan any template and see if there are 
suspicious/incorrect code patterns in them, so the templates will remain 
valid even after foreman code changes.

Recently I have started to think about user created templates and foreman 
upgrades.

When user upgrades foreman, hist default templates get upgraded by the 
installer/migrations, but templates created by the user (both cloned and 
from scratch) are not touched.
This could lead to invalid templates and broken provisioning functionality 
for the user.
Good example for this would be the change 

 
from calling to <%= foreman_url %> to <%= foreman_url('built') %> 

I was looking for a way to inspect any template, in order to identify 
problematic code as soon as the system is upgraded.

I came down to a solution based on rubocop - it's already analyzing source 
files for patterns.
I have created a POC that analyzes a template written to a file, and 
presents the resulting errors as regular rubocop (clang style).
All source codes are available as gist: 
https://gist.github.com/ShimShtein/341b746f15826261053e97c2f435ff1a

Small explanation for the gist:

Entry point: inspect_template.rb
Usage:
Put everything from the gist to a single folder and execute:

> inspect_template /path/to/template_source.erb

This script aggregates all the parts that are needed to create the 
clang-like output.

The process:

   1. Strip all non-ruby text from the template. This is done by 
   erb_strip.rb. It turns everything that is not a ruby code into spaces, so 
   the ruby code remains in the same places as it was in the original file.
   2. Run rubocop with custom rules and erb monkey patch and produce a json 
   report
  1. foreman_callback_cop.rb custom rule file. The most interesting 
  line is "def_node_matcher :foreman_url_call?, '(send nil :foreman_url)
  '". Here you define which pattern to look for in the AST, in our case 
  we are looking for calls (send) for foreman_url method without parameters.
  2. foreman_erb_monkey_patch.rb file: Patches rubocop engine to treat 
  *.erb files as source files and not skip them.
   3. Process the resulting json to convert it to clang style highlighting.

Possible usages:

   - Scanning all template after foreman upgrade to see that they are still 
   valid.
   - Linting a template while while editing.
   - Using rubocop's autocorrect feature to automatically fix offences 
   found by this process.

Long shot: we can create custom rules to inspect code for our plugins too, 
especially if we start creating custom rules in rubocop.

I am interested in comments, opinions and usages to see how to proceed from 
here.

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Proposal to move "How to create a plugin" page to github.

[sshtein]

Timo, 
Since we already are using the site to host the handbook, I think it would 
be nice if plugin writing guidelines would be there too.
If we look at plugin writers as "core users", would the site be better 
suited for the task? ;)

For the development workflow:
First, if the docs PR is mentioned in the code PR (as it should), Github 
makes it pretty easy to follow the link for the code maintainer.
IMHO there are two advantages for different repos
1. The docs PR is exposed to additional maintainers that could give 
additional input about the quality of the documentation itself (like 
styling, english e.t.c).
2. It decouples the lifecycles of the changes. Let's say that the code is 
already perfect, but the docs still need some work. I don't think we should 
not merge the code and wait for the docs to mature.

Marek, 
I think a simple link in our readme file would be a quite good compromise 
for syncing code and docs repos. I suggest either "Plugins" or "How to 
contribute" chapters. It should also increase the visibility of the page. 
+1 to google optimization
About template writing wiki: I don't want to derail this thread, but after 
we agree on a place for plugin writing wiki, I can start another one for 
template writing as well. One issue per commit, isn't it ;)



On Friday, June 23, 2017 at 9:36:56 AM UTC+3, Marek Hulán wrote:
>
> > Am 22.06.17 um 11:16 schrieb Greg Sutcliffe: 
> > > +1 for moving the plugin documentation to the plugins part of the 
> > > website. I think that makes sense, and we can (as with other areas) 
> > > keep the wiki for footnotes, edge cases, and tips. 
> > > 
> > > Greg 
> > 
> > I'd suggest to add a Markdown file to the core repo. I believe code 
> > documentation is better suited to be in the code repo and the 
> > documentation can be reviewed alongside the code change in core by core 
> > maintainers. The website should be more for foreman users. But 
> > theforeman.org is fine with me as well. 
> > 
> > - Timo 
>
> Having this info in the codebase itself sounds reasonable but I'd prefer 
> having it on one place with other pages such as template writing wiki. The 
> later is also useful for users and ideally will be moved to theforeman.org 
> one 
> day. 
>
> If there was a reliable way to sync it from theforeman.org to core 
> codebase, 
> that would we great. If I had to choose between these two, then I find 
> theforeman.org better. I think Google would give better results for 
> potential 
> plugin writers too. 
>
> -- 
> Marek 
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Proposal to move "How to create a plugin" page to github.

[sshtein]

Hi all,

Recently I have created a couple of PR's that added extension point to 
plugins. These extension point needed to be documented in the 
http://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Plugin
 document.
The problem is that between the time I have created the code PR and the 
time it got merged (and the doc change was necessary) passed a couple of 
months. I had to remember the context when I was writing the PR in order to 
write the wiki page.

I suggest creating a docs PR side by side with the code PR and creating a 
reference between them.
Benefits:
1. The documentation would be created in the same context by the developer, 
while it's fresh in his memory and he remembers all the caveats for the 
user.
2. The reviewers could better understand what the PR is offering.
3. There is less chance for the developer to forget to update the docs

In order to do that, we have to move the page to github (somewhere in 
theforeman.org  repo).
To mitigate a concern about the ease of change, we can put a link to edit 
the page in the header/footer of the page itself, so it will be one click 
away for anyone who wants to edit it.

Comments?

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Performance analysis deep-dive, what would you like to hear about?

[sshtein]

Hey, I want to do a performance analysis deep-dive.

What I had in mind:

1. Installation of tooling in development environment
2. Simple request breakdown (view, sql)
3. Debugging API request
4. Time graph (flamegraph)
5. Memory consumption
6. Installation in prod env
7. Analyzing POST requests

Feel free to ask questions and suggest topics that you want me to cover.

Shim.

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Rubocop enhancements for foreman plugins - your opinion is needed

[sshtein]

As we all know, foreman core uses rubocop for enforcing style rules. Many 
plugins, especially those that are in theforeman organization, use rubocop 
too.
The problem is, that the rules are not unified between foreman core and 
plugins. In addition to that when rubocop version changes in core, core's 
rules change accordingly, but plugins remain often with the old ruleset for 
no reason.

I am suggesting inheriting foreman's ruleset as a base ruleset for plugins.
I have found two ways to accomplish that:

   1. Rubocop has an option to inherit a remote URL [1] 
   
,
 
   so we can point a plugin to github URL for foreman master ruleset [2] 
   . 
   
   Advantages: it's simple. 
   Disadvantages: You have to be connected in order to download the file 
   (Although rubocop has caching mechanism for it)
   2. We can utilize another Rubocop config option: inheriting the settings 
   from a gem [3] 
   
.
 
   This will require us to create a special development gem, that would be 
   used by foreman core and plugins. This gem will contain a proper ruleset.
   I am a bit biased, since I have already started foreman_devel plugin [4] 
    that should help plugin 
   developers in multiple tasks.
   Advantages: It's a plugin that can do a lot more than just ruleset repo. 
   It's versioned properly. It's offline, once you have the gem installed.
   Disadvantages: Extra gem. Installation is more complicated. Affects the 
   core too

I would like to hear your opinions about the issue. Both whether you like 
the basic idea of inheriting the same ruleset and if so, which is the 
preferred way to go.

Shim,



[1] 
https://github.com/bbatsov/rubocop/blob/master/manual/configuration.md#inheriting-configuration-from-a-remote-url
[2] 
https://raw.githubusercontent.com/theforeman/foreman/develop/.rubocop.yml
[3] 
https://github.com/bbatsov/rubocop/blob/master/manual/configuration.md#inheriting-configuration-from-a-dependency-gem
[4] https://github.com/ShimShtein/foreman_devel

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Re: New community-templates structure

[sshtein]

For navigation (reading) IMHO kind/subkind works well (although I have to 
admit I am not a sysadmin - I didn't have to do it as my day job)
What about modification use case?
Is there any use case where a user wants to modify multiple templates from 
different kinds? For example something along the lines of "I want to modify 
all my kickstart templates".



On Friday, November 25, 2016 at 4:41:01 PM UTC+2, Marek Hulán wrote:
>
> Hello foreman devs, 
>
> As I demonstrated on last community demo [1] templates can be now easily 
> exported from Foreman. I also mentioned that I'm working [2] on 
> foreman_templates feature to easily export all templates to a given git 
> repo. 
> Since the plugin can be used with any git repo, not just the community- 
> templates [3] one, I'd like to standardize the format of such repository. 
>
> When we export templates from Foreman, we can only use template attributes 
> to 
> determine the resulting path in the repo. First obvious idea is to name 
> the 
> template file according to it's name in Foreman. That's probably not 
> enough 
> since it wold result in one directory mixing all partition tables, 
> provisioning templates and job templates. So I came up with structure like 
> $template_type/$name. For better look and feel of what it means, you can 
> see 
> it in one of my branches [4]. 
>
> Currently we also separate provisioning templates into more directories. I 
> don't think the rule is well defined today. I think we could separate it 
> per 
> template kind. Again you can see this demonstrated in another branch at 
> [5]. 
> It applies only to provisioning templates, other types don't have template 
> kind attribute. 
>
> Please share your ideas for other structuring or which of schema mentioned 
>   
> above you find better. The level of nesting does not matter from technical 
> point of view but I think 2 or 3 directories is the limit. 
>
> The ultimate goal of making foreman_templates exporting compatible with 
> community-templates is making sharing of user changes easy, in fact just a 
> matter of opening PR from the forked repo. Another nice benefit would be 
> that 
> future changes in metadata, e.g. adding organizations and locations keys 
> would 
> be much easier, we'd just reexport all templates from Foreman with updated 
> export code. 
>
> Since we now have metadata as a part of each template we could also 
> improve 
> seeding to avoid hard coding the list in seed files [6] 
>
> [1] https://www.youtube.com/watch?v=M0-3x8AUfFQ 
> [2] https://github.com/theforeman/foreman_templates/pull/36 
> [3] https://github.com/theforeman/community-templates 
> [4] https://github.com/ares/community-templates/tree/develop_kind_only 
> [5] 
> https://github.com/ares/community-templates/tree/develop_kind_and_subkind 
> [6] 
> https://github.com/theforeman/foreman/blob/develop/db/seeds.d/07-provisioning_templates.rb#L21-L94
>  
>
> Thanks for all comments 
>
> -- 
> Marek 
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Using tmpfs significantly reduces testing time

[sshtein]

I have ext4 for my home and tmpfs for /tmp.
I haven't tried your solution (symlink), but if you say it's pretty much 
the same as in-memory, I will surely switch to it.


On Wednesday, August 24, 2016 at 12:10:05 PM UTC+3, Lukas Zapletal wrote:
>
> > It would be nice to see the differences. 
>
> It's almost the same speedup as with symlink (the difference is loading 
> the database three times perhaps): 
>
> real13m32.018s 
> user11m13.229s 
> sys 0m12.004s 
>
> What is your file system? That might do the difference. 
>
> -- 
> Later, 
>  Lukas #lzap Zapletal 
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Using tmpfs significantly reduces testing time

[sshtein]

I have manged to hack the memory DB to work [1], but for me, the time 
difference wasn't that big (~10%).

It would be nice to see the differences. 

[1] 
https://github.com/ShimShtein/foreman/compare/15846...ShimShtein:memory_try?expand=1



On Monday, August 22, 2016 at 2:44:07 PM UTC+3, Lukas Zapletal wrote:
>
> > > I heard that there is an option to use sqlite in-memory. 
> > > I gave to admit I didn't try it, but I think it worth mentioning. 
> > 
> > first google hit: 
> > 
> http://www.calicowebdev.com/2011/01/25/rails-3-sqlite-3-in-memory-databases/ 
>
> Those first hits :-) I have considered that of course, it does not work. 
> The moment connection is closed the database dies. Migration, db load 
> and test rake tasks are executed as separate processes. 
>
> -- 
> Later, 
>  Lukas #lzap Zapletal 
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Re: Host inheritance of hostgroup values

[sshtein]

Hi,
First I want to make sure what version of foreman do you have and are you 
using Katello plugin? Especially do you have the "inherit" button on the 
environment field.
If you do - you have two choices: one would be to set the environment field 
in the server side ( create a filter method and add it to HostsController 
class) you can see an example here 

.
Second choice would be to do it on client side - you can mimic the way the 
property is marked when it's set by the server: You can add "data-explicit" 
attribute to the field. 

You can find the JS that manipulates this field here 

.


On Saturday, July 9, 2016 at 11:56:13 PM UTC+3, Matthew Ceroni wrote:
>
> Is there a way to prevent the host from inheriting a host group value if 
> it is not set?
>
> I have written a small plugin that sets the environment based on the 
> hostname. However, if I then select the hostgroup it resets that 
> environment to blank because the hostgroup doesn't have a value. 
>
> I have tried everything via javascript to get around this. I set an 
> onsubmit function to set the environment (again based on hostname) but this 
> onsbumit function gets over-ridden (I think) when the hostgroup is 
> selected. 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.