Re: [foreman-dev] Proposal: Let's make notifications optional

[Lukas Zapletal]

> 2. do you know that there is a global env already? just run:
>
> NOTIFICATIONS_POLLING=100 foreman start

I hereby withdraw my proposal, this works for me. Quite hidden one :)

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Review of all our logrotate configuration

[Lukas Zapletal]

Hey,

I spent some time reviewing logrotate scripts because we had some
reports on logrotation when SELinux was preventing it. TL;DR -
resolved the issue, filed temporary fix for our policy and RHEL BZ.

Comments below:

/var/log/foreman/*.log
Works ok, copy and truncate.

/var/log/foreman-proxy/proxy.log
/var/log/foreman-proxy/smart_proxy_dynflow_core.log
Logrotate sends SIGUSR1 which is prevented by SELinux, tracked under
http://projects.theforeman.org/issues/19053

/var/log/puppet/*log
Similar problem but SIGUSR2

For both above, I filed a RHEL BZ
https://bugzilla.redhat.com/show_bug.cgi?id=1527522 and I filed a
temporary PR to solve this via our own policy rule until RHEL 7.6 is
out. https://github.com/theforeman/foreman-selinux/pull/75

Candlepin - had some issues in the past, these are solved now.

Pulp - logs into syslog, no rotation needed.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Pull request github template proposal

[Lukas Zapletal]

I created the following proposal:

https://github.com/theforeman/foreman/pull/5099/files

I tried to keep it short and simple. Please comment.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Re: Foreman instrumentation for telemetry proposal

[Lukas Zapletal]

The PR is finally up for review.

https://github.com/theforeman/foreman/pull/5096

I recorded short demo how to set it up with Prometheus via statsd:

https://youtu.be/i5iCOLEByZk

On Mon, Nov 20, 2017 at 3:46 PM, Lukas Zapletal <l...@redhat.com> wrote:
> Hey,
>
> on the last demo I presented my proposal for telemetry (it is actually
> a separate video). I am looking for non-intrusive approach with broad
> integration possibilities:
>
> https://www.youtube.com/watch?v=gCLSI9-4QpE
>
> This was also showed on our demo last week (the same content):
> https://www.youtube.com/watch?v=QHzNIFjMpTM
>
> I am starting this thread to gather feedback before I open a PR with
> this. Currently the code is mostly in Rails initializer and looks like
> this:
>
> # get telemetry singleton instance and setup it
> telemetry = Foreman::Telemetry.instance.setup(... some options ...)
>
> # register measurements
> telemetry.add_counter(:http_requests, 'A counter of HTTP requests
> made', [:controller, :action])
> telemetry.add_histogram(:http_request_total_duration, 'Total
> duration', [:controller, :action])
> telemetry.add_counter(:activerecord_instances, 'Number of instances of
> AR models', [:class])
>
> # send measurements from Rails instrumentation or from code base
> telemetry.increment_counter(:http_requests, 1, :controller =>
> controller, :action => action, :status => status)
> telemetry.observe_histogram(:http_request_total_duration, duration,
> :controller => controller, :action => action)
>
> The proposed API is a single class (a singleton actually) with three
> registering methods and three measure methods. I don't think such a
> simple class needs proper separation of concerns, but we can talk
> about this in the PR. The registration part could be turned into some
> kind of DSL, currently it takes metric name, description and list of
> keys which will be part of an instance for those frameworks which do
> not support arbitrary amount of key-value pairs.
>
> If there are no objections, I will add settings and better error
> handling and file the PR.
>
> --
> Later,
>   Lukas @lzap Zapletal



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] DHCP filename option no longer honored

[Lukas Zapletal]

Thanks a bunch, will test and merge.

I already checked 1.16/1.15 this is develop only.

LZ

On Fri, Dec 15, 2017 at 7:08 AM, Timo Goebel <m...@timogoebel.name> wrote:
> Hi,
>
> Dimitri sent a patch that should fix the problem. It's just a one line
> change, so we just need to test it and it should be ready to merge.
>
> However, we should check what change really introduced the problem to see if
> this needs backporting to 1.16. I have some doubts it was the change
> mentioned below.
>
> Lukas, can you please confirm the patch fixes the issue for you?
>
> Timo
>
> Am 14.12.17 um 14:21 schrieb Lukas Zapletal:
>>
>> Hey,
>>
>> I noticed that DHCP filename option is no longer honored in
>> develop/1.17 branch anymore. This is regression from 1.15/1.16 series,
>> filed a blocker bug for 1.17:
>>
>> http://projects.theforeman.org/issues/21975
>>
>> This breaks UEFI provisioning. I git-bisected
>>
>> commit e75df1dd3cc30bff2832d337b5bc20bd822e209a
>> Refs:
>> Author: Timo Goebel <m...@timogoebel.name>
>> AuthorDate: Wed Oct 25 16:26:01 2017 +0200
>> Commit: Dmitri Dolguikh <witlessb...@gmail.com>
>> CommitDate: Mon Nov 6 14:28:09 2017 -0800
>>
>> But I cannot figure it out, help needed. Thanks.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman instrumentation for telemetry proposal

[Lukas Zapletal]

Exactly, I based my proposal on Prometheus and Statsd. You can choose.

I am going to work on my PR next week hopefully, but you can test it today:

https://gist.github.com/lzap/2dfdd4dea29786a837cb1b7feb7862fd

Thanks for feedback!

On Thu, Dec 14, 2017 at 3:52 PM, Tom McKay <thomasmc...@redhat.com> wrote:
> Ooops! I should have watched your video first. :) Watching it now. "Proposal
> to integrate Prometheus and Statsd instrumentation libraries into Foreman
> ..."
>
> On Thu, Dec 14, 2017 at 9:26 AM, Tom McKay <thomasmc...@redhat.com> wrote:
>>
>> Openshift uses Prometheus[1] which seems very similar and compatible with
>> your ideas. Is that something you've looked at already? If/when foreman is
>> containerized and perhaps run under kubernetes your work could be very
>> useful as well.
>>
>> https://blog.openshift.com/tag/prometheus/
>>
>>
>>
>> On Fri, Nov 24, 2017 at 12:54 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>
>>> > 1. what happened to the PCP approach we talked about in the past?
>>>
>>> Thats going in parallel, PCP is just a monitoring framework you can
>>> integrate with instrumentation data just like any other.
>>>
>>> > 2. how would you integrate this to sosreport/foreman-debug? I'm
>>> > thinking of
>>> > storing the statsd data locally, collecting them with foreman-debug,
>>> > and
>>> > then, being able to import them later to the prometheus and other
>>> > tools. Is
>>> > this how this could work? Any other options?
>>>
>>> This is my ultimate goal to have working PCP deployment including
>>> telemetry data and archives could be collected by foreman-debug, they
>>> are pretty small (few MBs per day).
>>>
>>> > 3. does every host/runtime needs it's own statsd service, or there
>>> > would be
>>> > one shared process? Asking bith for multi-host and containers use-case
>>>
>>> It is up to you if you want one statsd service per guest/container,
>>> host or subnet. Prometheus endpoint will not require any external
>>> daemon once sharing metrics is merged into upstream. For this reason,
>>> statsd will server as a temporary solution and alternative for the
>>> future.
>>>
>>> > The proposal of the telemetry api itself seems reasonable, let's
>>> > discuss
>>> > that on an actual PR
>>>
>>> Thanks, I hope to finish it this year.
>>>
>>> --
>>> Later,
>>>   Lukas @lzap Zapletal
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] The package.json thing in git

[Lukas Zapletal]

I have this aliases:

alias bi="bundle install && test -f package.json && npm install ||
node_modules/.bin/npm install"
alias bu="bundle update && test -f package.json && npm upgrade ||
node_modules/.bin/npm upgrade"

We don't have Gemfile.lock in git, therefore update needs to be
executed often. I guess it does not apply to npm. Confusing.


On Thu, Dec 14, 2017 at 4:24 PM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> On Thu, Dec 14, 2017 at 04:18:45PM +0100, Lukas Zapletal wrote:
>>
>> for some reason, npm upgrade is touching my package.json on every run.
>> This file is in git, so it is annoying to have that edited every run.
>> Can we do something with this?
>
>
> I think you're supposed to use npm install. npm upgrade is an alias for npm
> update (don't get me started that there's also the hardcoded udpate alias).
> npm update updates a package.
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] DHCP filename option no longer honored

[Lukas Zapletal]

Hey,

I noticed that DHCP filename option is no longer honored in
develop/1.17 branch anymore. This is regression from 1.15/1.16 series,
filed a blocker bug for 1.17:

http://projects.theforeman.org/issues/21975

This breaks UEFI provisioning. I git-bisected

commit e75df1dd3cc30bff2832d337b5bc20bd822e209a
Refs:
Author: Timo Goebel 
AuthorDate: Wed Oct 25 16:26:01 2017 +0200
Commit: Dmitri Dolguikh 
CommitDate: Mon Nov 6 14:28:09 2017 -0800

But I cannot figure it out, help needed. Thanks.


-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] CDN for the theforeman.org

[Lukas Zapletal]

Great news, thanks.

LZ

On Tue, Dec 12, 2017 at 4:12 PM, Evgeni Golov <evg...@redhat.com> wrote:
> downloads.theforeman.org and stagingdeb.theforeman.org is now served
> via Fastly, enjoy!
>
> On Tue, Dec 12, 2017 at 3:51 PM, Evgeni Golov <evg...@redhat.com> wrote:
>> Just a quick heads-up for everybody.
>>
>> We now have the right config and TLS certificates in place to serve
>> Forman content via Fastly, but we did not change any public DNS
>> entries yet.
>> We'll probably start with stagingdeb.tfm.o and downloads.tfm.o soon,
>> and move to the other vhosts some time in January if no problems
>> occur.
>>
>> Evgeni
>>
>> On Mon, Nov 20, 2017 at 1:24 PM, Evgeni Golov <evg...@redhat.com> wrote:
>>> FWIW, deb.d.o runs on a custom LE cert, not on a wildcard, so you
>>> can't really compare that.
>>> And cdn-fastly.deb.debian.org (which is the "real" Fastly host apt
>>> hits) does not have SSL enabled at all.
>>>
>>> On Mon, Nov 20, 2017 at 11:58 AM, Greg Sutcliffe
>>> <g...@emeraldreverie.org> wrote:
>>>> On 20/11/17 10:08, Lukas Zapletal wrote:
>>>>> Great, thanks. Before moving on, I'd check if yum/deb accepts these
>>>>> kind of certificates. I *think* so, but I've heard from security guys
>>>>> they don't like these certs at all :-)
>>>>
>>>> Thanks for the heads up, good to know. I believe Fastly is used by the
>>>> Debian repos themselves, so I *think* it's OK, but we can check. We can
>>>> always choose not to use their certificate if we want, we have
>>>> LetsEncrypt setup on that node anyway...
>>>>
>>>> Evgeni and I will take a look at this today - Neil, maybe I'll ping you
>>>> later in the week if we get stuck :)
>>>>
>>>> Greg
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups 
>>>> "foreman-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>>> --
>>> Beste Grüße/Kind regards,
>>>
>>> Evgeni Golov
>>> Software Engineer
>>> 
>>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
>>> Commercial register: Amtsgericht Muenchen, HRB 153243,
>>> Managing Directors: Charles Cachera, Michael Cunningham, Michael
>>> O'Neill, Eric Shander
>>
>>
>>
>> --
>> Beste Grüße/Kind regards,
>>
>> Evgeni Golov
>> Software Engineer
>> 
>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
>> Commercial register: Amtsgericht Muenchen, HRB 153243,
>> Managing Directors: Charles Cachera, Michael Cunningham, Michael
>> O'Neill, Eric Shander
>
>
>
> --
> Beste Grüße/Kind regards,
>
> Evgeni Golov
> Software Engineer
> 
> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243,
> Managing Directors: Charles Cachera, Michael Cunningham, Michael
> O'Neill, Eric Shander
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Custom hiera in Katello 3.4 deployment

[Lukas Zapletal]

Hey,

I have Katello 3.4 and I created this file:

/etc/foreman-installer/custom-hiera.yaml

---
dns::zones:
  home.lan:
soa: winter.home.lan
soaip: 192.168.99.1
contact: hostmaster.home.lan.

After re-running the installer (in dry mode) nothing really happens.
This is already configured system, is there anything I need to do in
order to add this zone?

Thanks

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Nominating additional maintainers in foreman-core

[Lukas Zapletal]

+1 +1 beer time!

LZ

On Wed, Dec 6, 2017 at 1:40 PM, Tomer Brisker  wrote:
> Hello,
> As many of you may have noticed, foreman-core open PR count has been in the
> area of 100-110 for most of the last few months. There are only a few people
> with merge access that actively review PRs, so that the time it takes for
> changes to be accepted can take long. I would like to propose granting merge
> access to two more members of the community:
>
> Shimon Shtein - has 75 merged commits [1] and has taken part in the reviews
> of 64 merged commits[2].
> Michael Moll - has 71 merged commits [3] and has taken part in the reviews
> of 130 merged commits[4]. He is also already a long time maintainer of
> several of our other repos and has proven to be a responsible maintainer.
>
> Please send +1/-1 either to the list or to me in private if you prefer.
> Per our process, unless I receive any objections within a week, I will
> request one of the organization owners to grant them access.
>
> [1]
> https://github.com/theforeman/foreman/pulls?q=is%3Apr+author%3Ashimshtein+is%3Aclosed
> [2]
> https://github.com/theforeman/foreman/pulls?utf8=%E2%9C%93=is%3Apr+-author%3Ashimshtein+commenter%3Ashimshtein+is%3Aclosed+
> [3]
> https://github.com/theforeman/foreman/pulls?utf8=%E2%9C%93=is%3Apr+author%3Ammoll+is%3Aclosed+
> [4]
> https://github.com/theforeman/foreman/pulls?utf8=%E2%9C%93=is%3Apr+-author%3Ammoll+commenter%3Ammoll+is%3Aclosed+
>
> --
> Have a nice day,
> Tomer Brisker
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Is it time to remove turbolinks?

[Lukas Zapletal]

The idea behind turbolinks is good in general - it works great when
you have a standard Rails application with its view stack (ERB). When
I saw this talked, I liked it very much:

https://www.youtube.com/watch?v=SWEts0rlezA

Unfortunately, we are not good fit. This works for a clean app which
ERB and maybe little javascript. Foreman (and Katello) is a heavy JS
app with various JS frameworks and perhaps it does not work well. At
least this is my impression, I am very far from being a frontend
developer :-)

LZ

On Wed, Dec 6, 2017 at 4:10 PM, Walden Raines  wrote:
> I am seeing more and more issues [1][2] around turbolinks and I'm wondering
> if it's time to remove it from foreman.  I have seen it recommended that one
> shouldn't use the back button in foreman; I thought that request was
> reserved for poorly written php sites that resubmit forms when the back
> button is pressed, we're better than that! :)
>
> Are the benefits of turbolinks worth the cost?  Does anyone have any metrics
> on the difference in page load times with and without "turbo"links?
>
> Any additional thoughts on its removal?
>
> Cheers,
> Walden
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1522740
> [2] http://projects.theforeman.org/issues/21480
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] RFC: File audit for template rendering

[Lukas Zapletal]

You are right that in general, this is an anti-pattern, but the data
we will be sending is write only logging data. It's essentially a log
sink, but for arbitrary data (blobs, files, name it you want). In
clustered setup, you simply keep the logs on nodes along with other
logging data. On the other hand, we aim at using syslog for everything
in the future, this could be additional thing to keep in mind.

We also discussed storing this in database schema in a formal way (new
table rendered_template with references to host/template_kind) but
arguments against are this is already quite complex part of our
schema, keeping history could easily grow the database and performance
could be problem for Remote Execution when performing large-scale
script runs. We ended up discussing actually removing audit trail data
from database, but this turned to be problem as we need to keep
reference to taxonomy, users and records.

This brings me to alternative proposal:

I plan to work on full syslog support for Rails and Smart Proxy in
January next year. The goal is to achieve structured logging using
some standard format (CEE/CEF). We would send those rendered templates
to Rails log (syslog) and there could be an optional syslog module to
extract those log entries and put them into similar directory
structure. This could be the default setup for small deployments, but
for large-scale we will recommend to send logs over network to central
logging point anyway, so the module could be installed there or
depending on setup this could be loaded into ELK stack. (*)

In short, let's send the rendered templates to syslog, it deals with
multiple lines just fine, we can simply filter these from
production.log and put them into a separate file or even directory
structure.

(*) http://www.rsyslog.com/writing-rsyslog-plugins-in-2-minutes/

On Tue, Dec 5, 2017 at 6:34 PM, Timo Goebel <m...@timogoebel.name> wrote:
> I don't think, this is a good idea. Storing files on the local filesystem is
> generally consideres an anti-pattern imho. Both for web applications and
> expecially in a container world.
>
> In a clustered setup, you'd have to attach some kind of shared-storage to
> the foreman hosts to make this work. Possibly via NFS or something. This is
> meant to cause trouble. I generally advise not to have any state on an
> application server.
>
> I think, this can and should be done with a database. I agree, that the
> audit model might not be the best place for this. But a database definitely
> is.
>
> Just my 2 cents.
>
> - Timo
>
> Am 05.12.17 um 18:01 schrieb Lukas Zapletal:
>
>> Hello,
>>
>> Foreman does have a nice audit trail for many operations, what's
>> missing is ability to find how a template (e.g. kickstart) was
>> rendered. Storing whole template text in audit table is probably not
>> the best thing to do, production.log is also not a good fit, so here
>> is a proposal.
>>
>> I want to create a small API called File audit (*) with ability to
>> store arbitrary files into
>> /var/log/foreman/file_audit/id/text-timestamp where "id" is record id
>> (or multiple ids concatenated with dots), "text" is arbitrary alphanum
>> text and "timestamp" is unix epoch number. Example:
>>
>> /var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603
>>
>> That API will be used to store contents of all templates rendered, so
>> users can easily go "back in time" and see how templates are being
>> rendered. The directory would be root-only reading and files will be
>> created with restricted permissions (foreman user rw only). On system
>> with SELinux, security would be more tightened allowing writing only
>> to foreman_t domain and reading to nobody. For the example above, this
>> would mean:
>>
>> 1.33.7/my.host.lan-pxelinux-1512492603
>>
>> 1 - file audit type (static list, 1 stands for "template audit entry")
>> 33 - host id
>> 7 - template id
>> my.host.lan-pxelinux - extra data so users can work and search from
>> command line
>> 1512492603 - UNIX epoch timestamp
>>
>> Everytime new record is added, a log entry is created into
>> production.log containing file path. By default, there will be a cron
>> job deleting all files older than one month. In documentation, we will
>> ask users to rsync the directory to different location if long-term
>> archival is needed.
>>
>> This API could be used for other audit logging, for example when user
>> uploads a manifest ZIP file in katello or new version of RPM/Puppet
>> file. This will be the first step to improve audit around templates,
>> later on we can create a plugin showing the data in audit/host pages
>&

[foreman-dev] RFC: File audit for template rendering

[Lukas Zapletal]

Hello,

Foreman does have a nice audit trail for many operations, what's
missing is ability to find how a template (e.g. kickstart) was
rendered. Storing whole template text in audit table is probably not
the best thing to do, production.log is also not a good fit, so here
is a proposal.

I want to create a small API called File audit (*) with ability to
store arbitrary files into
/var/log/foreman/file_audit/id/text-timestamp where "id" is record id
(or multiple ids concatenated with dots), "text" is arbitrary alphanum
text and "timestamp" is unix epoch number. Example:

/var/log/foreman/file_audit/1.33.7/my.host.lan-pxelinux-1512492603

That API will be used to store contents of all templates rendered, so
users can easily go "back in time" and see how templates are being
rendered. The directory would be root-only reading and files will be
created with restricted permissions (foreman user rw only). On system
with SELinux, security would be more tightened allowing writing only
to foreman_t domain and reading to nobody. For the example above, this
would mean:

1.33.7/my.host.lan-pxelinux-1512492603

1 - file audit type (static list, 1 stands for "template audit entry")
33 - host id
7 - template id
my.host.lan-pxelinux - extra data so users can work and search from command line
1512492603 - UNIX epoch timestamp

Everytime new record is added, a log entry is created into
production.log containing file path. By default, there will be a cron
job deleting all files older than one month. In documentation, we will
ask users to rsync the directory to different location if long-term
archival is needed.

This API could be used for other audit logging, for example when user
uploads a manifest ZIP file in katello or new version of RPM/Puppet
file. This will be the first step to improve audit around templates,
later on we can create a plugin showing the data in audit/host pages
if needed. But in the first phase, administrators could easily
search/grep/diff those files when necessary.

(*) if you have a better name, please do propose

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Speed up Foreman development booting by 50%

[Lukas Zapletal]

I just filed this PR which speeds booting up quite a lot, from 10s to
6s on my system:

https://github.com/theforeman/foreman/pull/5045

Feel free to test, I haven't tested with Katello but it should give
even bigger boost with it.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Firefox 57 and noVNC

[Lukas Zapletal]

This is Sat 6.3 install (Katello 3.4), no changes here:

# Websockets
:websockets_encrypt: on
:websockets_ssl_key: /etc/pki/katello/private/katello-apache.key
:websockets_ssl_cert: /etc/pki/katello/certs/katello-apache.crt

# SSL-settings
:ssl_certificate: /etc/foreman/client_cert.pem
:ssl_ca_file: /etc/foreman/proxy_ca.pem
:ssl_priv_key: /etc/foreman/client_key.pem

On Tue, Dec 5, 2017 at 12:06 PM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> On Tue, Dec 05, 2017 at 09:43:20AM +0100, Lukas Zapletal wrote:
>>
>> I was trying noVNC the other day on FF 57. Imported katello CA (it was
>> a katello intance), but it didn't work until I flipped flag
>> "network.websocket.allowInsecureFromHTTPS" on and then it worked.
>>
>> I remember that Firefox always worked out of box - when server CA was
>> imported, it worked like charm. In the documentation we have "FF might
>> need to turn on this flag". Anyone knows under which circumstances we
>> need to flip this on?
>>
>> Any luck setting up FF 57 without any hacking with noVNC?
>>
>> https://theforeman.org/manuals/1.15/index.html#7.1NoVNC
>
>
> There are settings (websockets_ssl_{key,cert}) to choose which certificate
> is presented by the VNC websockets proxy. Are you sure the correct
> certificates are used by it?
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Koji Space on /

[Lukas Zapletal]

My 2C:

1) We need to setup foreman-infra mailing list, we had one set by
Dominic but I am not sure we still have access to this. Greg, can you
set up something for us and add our infra team? Whatever this is, but
personally I'd prefer emails :)

I had an email about this on Saturday but I haven't read it until
today. I am bad in this.

2) By default httpd rotation has delayed compression so on busy day,
it can create one or two huge files until this is compressed. Although
we can tune logrotate, I've done this:

I moved and symlinked the following directories to local ephemeral volume:

* /var/cache/mrepo -> /mnt/tmp/cache/mrepo
* /var/log/httpd -> /mnt/tmp/log/httpd

Restarted httpd making sure there are no open jobs. Everything is fine
now, report issues.

On Mon, Dec 4, 2017 at 4:25 PM, Eric D Helms  wrote:
> Today Koji filled up its root partition which is designed to be small. In
> part, there was 2.0 GB of httpd logs due to the API intensive use of Koji.
> These logs are currently rotated and not compressed in anyway.
>
> How should we handle this? A few ideas for discussion:
>
>  1) Modify logrotate configuration to try to compress the logs
>  2) Setup a cron job to compress the rotated logs
>  3) Rotate the logs less
>
> We also noticed about 850MB of mrepo cache.
>
> --
> Eric D. Helms
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Firefox 57 and noVNC

[Lukas Zapletal]

Hello,

I was trying noVNC the other day on FF 57. Imported katello CA (it was
a katello intance), but it didn't work until I flipped flag
"network.websocket.allowInsecureFromHTTPS" on and then it worked.

I remember that Firefox always worked out of box - when server CA was
imported, it worked like charm. In the documentation we have "FF might
need to turn on this flag". Anyone knows under which circumstances we
need to flip this on?

Any luck setting up FF 57 without any hacking with noVNC?

https://theforeman.org/manuals/1.15/index.html#7.1NoVNC

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Request to Releasers: Please pull in translations and merge strings

[Lukas Zapletal]

Thanks for reminder, I am so bad in this.

To github.com:theforeman/foreman_discovery.git
   d44e91f..40f44dc  develop -> develop

LZ

On Sat, Dec 2, 2017 at 5:54 PM, Bryan Kearney  wrote:
> This is a request for folks who release plugins, either foreman or hammer.
>
> As part of your release process, please make sure that you are doing the
> following:
> 1) Pulling in the latest translations from transifex
> 2) Extracting strings from your code, and merging in those translations
>
> Almost all the plugins have a Makefile which will do this for you. If
> you got a pull request from me on 1 December, I likely added one or
> fixed it. I am happy to answer questions on how to use it, and to help
> folks in getting their stuff into transifex (/me is looking at you
> foreman-maintain folks).
>
> I have a process which will take translations from downstream and put
> them into transifex. This process ensures the community gets the value
> of the downstream translators. But, it requires that each time a release
> is done the builder refreshes their strings and pulls in the latest
> translations.
>
> -- bk
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Unique IDs and UI testing automation

[Lukas Zapletal]

Just to be sure it does not get lost:

> All inputs, including those not technically in a form (row select
> checkboxes, for example)
> All buttons

All buttons (which are links) already has data-id which is always
present and when not, it is auto-generated from HREF attribute. This
was added by Og request, perhaps it did not serve well?

Autogeneration cannot be used in other cases, I guess there is no
other way than explicitly adding those IDs manually.

I vote for raising an exception when ID is missing and Rails are in
development mode so we will never add such an element without an ID.
Would be good to have some kind of ID "registry" also available for
plugins so we prevent from making double IDs. I can imagine something
similar than what's used in i18n world: N_(id) when you can easily
create simple parser to verify whole codebase to extract those strings
(and easily spot wrong IDs).

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Unique IDs and UI testing automation

[Lukas Zapletal]

This code was actually created by me to server the very same purpose -
QAs asked for this. They wanted to have all buttons (links) to have an
ID and now they have it.

https://github.com/theforeman/foreman/commit/bfee97b6a6980b6ad4f5bd65f4b60680f08967f5

This is the request: http://projects.theforeman.org/issues/3751

We used data-id to carry the information. It is autogenerated from
HREF path, if that's missing then its "aid_not_defined".

I am not sure what happened and why QAs are not using this anymore. If
this is not used, it should go away. Indeed it looks like a hack, but
I wanted to have a generic solution to the problem - QAs always come
to us and request "can you add ID here, and here", this was supposed
to solve this issue so they are unblocked until we add explicit IDs
there.

LZ

On Wed, Nov 29, 2017 at 11:45 PM, Tomer Brisker  wrote:
> While I don't mind adding more IDs, care should be taken to make sure there
> aren't any duplicate ids on the same page.
> We already add data-id attributes to all links in a bit of a hacky way [1],
> I'd be happy to get rid of this unreadable code (which i'm not sure is even
> used anywhere) and replace it with actual ids in a consistent manner.
>
> [1]
> https://github.com/theforeman/foreman/blob/develop/app/helpers/application_helper.rb#L6
>
> On Wed, Nov 29, 2017 at 9:06 PM, Walden Raines  wrote:
>>
>> > Seems reasonable to me. I don't really see a downside other than the
>> > work to add all the IDs.
>>
>> I don't think we should go through the entire application and add the IDs.
>> I think the way to tackle this is two-fold:
>>
>> Add IDs as we update functionality on pages that don't have these IDs
>> Encouraging QE to report missing IDs (and potentially open a PR to add
>> them) on troublesome pages
>>
>> The latter has already been agreed to, the former is in our hands to
>> remember to do.
>>
>> Cheers,
>> Walden
>>
>>
>> On Wed, Nov 29, 2017 at 1:31 PM, Ewoud Kohl van Wijngaarden
>>  wrote:
>>>
>>> On Wed, Nov 29, 2017 at 01:08:13PM -0500, Walden Raines wrote:

 In order to assist in UI automation, I would like to propose that we add
 unique IDs to the following elements:

   - Tables and table rows (s)
   - All inputs, including those not technically in a form (row select
   checkboxes, for example)
   - All buttons

 These IDs should all follow the same format, perhaps something like
 object.class-object.id-element.type.  For example, product-2-row and
 product-2-checkbox.

 Thoughts on this?  Anything that needs to be added or changed?
>>>
>>>
>>> +1 This was also the feedback I heard from QE. Sometimes they have to use
>>> xpaths to test which are more likely to break than IDs.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Have a nice day,
> Tomer Brisker
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Re: Proposal: Foreman 1.XX = 2.0 (a new one)

[Lukas Zapletal]

Ok, gmail did not refresh me the thread, I was reading old posts.

> D) I want to stick with 1.X, talk about feature-driven release
> management or simply decide later

It looks like majority want to continue the discussion, please carry on there.

There are some voices that vertical nav with Rails 5.x might be a good
reason to do 2.0, but Eric preferred also to include V1 drop.

Also one interesting remark was that we don't need to rush and it does
not make any huge difference if we switch 1.17 or 1.19, that's also
worth mentioning.

That's for the summary.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Re: Proposal: Foreman 1.XX = 2.0 (a new one)

[Lukas Zapletal]

I think there is misunderstanding and the thread deviated to 2.0
feature discussion. This is not core of my proposal, I explicitly do
not want to do feature talk. There will always be something that we
would like to have in version X.Y. What I propose is:

A) Let's do 2.0 instead 1.17 regardless of features
B) Let's do 2.0 instead 1.18 regardless of features
C) Let's do 2.0 instead 1.19 regardless of features
D) I want to stick with 1.X, talk about feature-driven release
management or simply decide later

And it's totally fair if you guys prefer to talk about features and
decide later. This is just my attempt to set the direction back, but
it looks like I already know where this is going anyways :-)

At least I tried and opened the discussion so we don't hit the wall of
"twenty"...

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Proposal: Foreman 1.18 = 2.0

[Lukas Zapletal]

> Bikeshedding about SemVer aside, I'm good with doing a 2.0 release in
> the near future, but *please* lets use it to deprecate / drop stuff we
> no longer want to maintain. Otherwise there's no real point to it.

I agree we can take this "opportunity" to drop some deprecated things
like V1 API, but I don't see many other things. We are pretty good in
deprecating things using our "two releases" rule which should be
followed no matter if we bump major version or not.

Let's not block 2.0 with any feature, I wrote the reasons, if we fit
in some deprecation work why not. But's let's agree on 2.0 timeframe
regardless of any planning.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Proposal: Foreman 1.18 = 2.0

[Lukas Zapletal]

Where exactly do we have any info about SemVer for core? I know about
plugins, but not core.

Core plugin API gets rarely broken, we've been only extending it in
the past (although there were some cases I think).

LZ

On Wed, Nov 29, 2017 at 12:01 PM, Greg Sutcliffe
<g...@emeraldreverie.org> wrote:
> On 29/11/17 10:45, Lukas Zapletal wrote:
>> Oh no, *everyone* is talking features. :-) This has really nothing to
>> do with features, because that can easily fall into "neverending"
>> category of what's big enough change or not.
>
> Well, *technically* we use SemVer (I know, I know...). For that, while
> features play a part, for me it's more about correctly calling out
> breaking changes. So if we're every going to deprecate a bunch of stuff,
> then we should call *that* 2.0. Otherwise it's just grandstanding ;)
>  *Technically* we should be including the internal plugin API in our
> SemVer and bumping the major version every time plugins need to
> update... but that might get silly :P
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] [RFC] Deprecate and move Github RFC repo content

[Lukas Zapletal]

Yeah I think I misinterpreted how this feature works. I thought its
just a discussion per project. This is per team I guess, as you
pointed out.

Ok I am migrating back to wiki unless there is some blocker raised.

LZ

On Wed, Nov 29, 2017 at 11:37 AM, Greg Sutcliffe
<g...@emeraldreverie.org> wrote:
> On 29/11/17 08:47, Daniel Lobato Garcia wrote:
>> If it helps, I think
>> https://github.com/blog/2471-introducing-team-discussions looks also
>> like a good tool for these kind of discussions, without any need for
>> a moderator. The discussions are restricted to people in the GitHub
>> team.
>
> TLDR: -1 to this from me, I'm afraid. Let me explain.
>
> Firstly, we have to ask if this is a *replacement* for other
> discussions (i.e. dev-list), which I don't think you're proposing, or
> in-addition-to dev-list. I'll cover both, just to be complete.
>
> If it's an *additional* channel, then it will die. This is exactly what
> happened to the RFC repo the first time - we added an extra place to
> discuss things, the discussion fragmented, and then the network effect
> dragged all the discussion back to the dev-list.
>
> In addition, it's *even* more fragmented than the RFC repo, because it's
> per-team - and we have 63 teams. Yes, there's a "members" team, but even
> that only has 56 people in it (the org has 71, so something is out of
> line there). Limiting discussion to a single team means many more places
> to check if you want to know what's being discussed (but aren't already
> a part of the discussion).
>
> It's also worse than RFC repo in the sense that it still requires a
> GitHub account to participate - but now you *also* have to be in the
> team too. It does respect team structure, but we don't currently nest
> teams, and propagating discussions between child teams (eg plugins)
> looks awkward. That means we get into small, silo'ed discussions that
> don't get enough feedback from the wider community. That's dangerous
> echo-chamber / groupthink ground.
>
> On the other hand, if it's a *replacement* for dev-list (bear with me
> here :P), then I don't see anything here which is better than the
> *other* replacement for dev-list on the table (that's Discourse, which
> may happen looking at the current poll). There we *centralise* the
> discussion instead of fragmenting it, but keep the wiki-like powers, can
> use @group notifications to alert people, and users can join in too if
> they wish. As a *replacement*, team discussions thus feels inferior.
>
> Either way, I'm not a fan. This is a case of knowing the purpose of your
> communication channels and picking *one* way to handle it that suits the
> purpose. Having >1 just fragments everything (I learned that the hard
> way with the RFC repo, sadly). At the very least, I'd ask that we finish
> sorting out Discourse before we start on "communication changes, round
> 2" - we may find the new tooling should be part of that process.
>
> On 29/11/17 08:55, Lukas Zapletal wrote:
>
>> Greg, can you turn it on in RFC repo so we can test this? If this
>> fails, we can move to wiki and deprecate that repo.
>
> It's not per repo, it's per team, and GitHub has already enabled it
> across the entire site. It can't be disabled as far as I can see, so you
> can try it in any team you're a member of.
>
> example: https://github.com/orgs/theforeman/teams/discovery/discussions
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Proposal: Foreman 1.18 = 2.0

[Lukas Zapletal]

Oh no, *everyone* is talking features. :-) This has really nothing to
do with features, because that can easily fall into "neverending"
category of what's big enough change or not.

We've heard that 1.17 is possible, yeah I do agree but only if that's
confirmed by the release engineer, not sure if branches or anything
has been created already. Also we might already commented on the
internet about "upcoming 1.17", at least I did, so might be safer to
do 1.18. On the other hand, it's pretty easy for anyone to figure out
2.0 is the next version...

LZ

On Wed, Nov 29, 2017 at 11:14 AM, Sean O'Keeffe  wrote:
> +1 to the idea. Although 1.17 is a good candidate in terms of features, lets
> give ourselves time to decided what should be deprecated. Anytime after 1.17
> is good with me!
>
> On Wed, Nov 29, 2017 at 10:10 AM, Greg Sutcliffe 
> wrote:
>>
>> Given 1.17 will have:
>>
>> * vertical nav
>> * rails 5.X
>>
>> I think this is a good candidate, if we're ever going to do it. It's
>> also a good chance to put the woes of 1.16 behind us :)
>>
>> 1.17 hasn't been branched, but it's soon. If we're going to do this,
>> we'll need to decide what should be deprecated.
>>
>> Greg
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Proposal: Foreman 1.18 = 2.0

[Lukas Zapletal]

What have Linus Torvalds and me in common? We both don't remember
numbers higher than ten.

I propose to follow Linux kernel versioning and do 2.0 instead of 1.18
for no other reason that it's just too high number and it's
approaching crazy 20. "One point eighteen" sounds crazy, I always mess
up with "One point eight". Frankly, I kinda lost track somewhere after
1.14. :-)

I'd like also to propose to avoid exceeding 10 in the future - someone
speak up as we will approach to 1.8. Or we can put this to release
wiki.

If we already have some infrastructure set for 1.18, I propose 1.19 to
be 2.0 and act now - let's define a plan what needs to be done in
advance. I can only think of version number in RedMine, but you tell
me here what's needed.

I really hope it's not just me messing around with higher numbers or
having problems with typing four characters instead of three many
times.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Missing github issues features (no migration)

[Lukas Zapletal]

Sup!

When we introduced RedMine, Github issues were very limited. Calm
down, this is just a review of what's missing from Github issues, I am
not planning to propose or do anything :-) Let me do quick review of
missing features as of today (fall 2017) and possible mapping to
issues:

REDMINE - ISSUE

Status = Label
Priority = Label
Assigned To = Assignee
Category = Label
Target version = Milestone
Difficulty = Label
Found in release = Label
Votes = Reaction (+1)
Related issue = Issue link
Issue type = Subject (e.g. [TRACKER])
Bugzilla link = ???

So theoretically, we could map mostly everything to labels. There are
two ways of creating those labels, free form or with prefix. The
latter would look like:

Status-NeedMoreInfo
Priority-Urgent
Cat-Hostgroups
Difficulty-Easy
FoundIn-1.16

That looks ugly, frankly I'd prefer free form, because we often only
set just few (if not none) of the flags - Found In Version is one of
the most important ones and that could be as simple as "In 1.16".

So basically, we are missing bugzilla link and private comments. Is
the BZ link item only informative, or is there some non-human
processing behind? I know there is a bot but does it need BZ link? It
could search comments for some token for the same thing. For private
issues we could use Bugzilla (usually security bugs only), there is no
solution other than pay them for private repo.

Or maybe I miss the main reason why we are not using Github issues at all?

I like github integration with PRs, speed and good reliability (only
few blackouts per year) and also new features like projects. On the
other hand, it's full commitment to something not under our control
(today we can easily move our git somewhere else, but we still loose
all PRs).

This email is just to discuss possibilities, I know that migration to
Github would be painful and even too expensive or perhaps technically
not doable (how to migrate so many tickets). It's a pitty that github
is now getting features it really needed.

I also really like gitlab which is packed with super nice features,
theoretically migration to something like that would be easier (open
source). On the other hand, we'd need to host this and one thing is
having redmine down for an hour, different thing is inability to push.
But this is definitely a possibility, we also have some know how
already running our internal instance.

*Sigh*

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Re: [foreman-dev] Kanban tools and redmine

[Lukas Zapletal]

> Projects in GitHub are also per-team,

Those Projects look great, but it's a good fit only when you use
Issues as well. Having RedMine + GH Projects is not much added value I
think.

Ivan: Those votes I proposed would only choose if to put a plugin to
prod instance or not of course. You would choose whatever you like.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Re: [foreman-dev] [RFC] Deprecate and move Github RFC repo content

[Lukas Zapletal]

Yeah, I also think this would make sense. It's basically a wiki with
discussion - ideal.

Greg, can you turn it on in RFC repo so we can test this? If this
fails, we can move to wiki and deprecate that repo.

--

LZ
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Kanban tools and redmine

[Lukas Zapletal]

Can you elaborate on why something does not make sense to track in
RedMine? I am not following, RedMine is a generic ticketing system and
you can create as many tickets as you want, e.g. Buy milk. It will
work. This is a dogma I do not take as an argument. What's relevant is
if you need to create private work items, but we do have a private
feature in RedMine and we use it from time to time. And frankly, we
don't need these very often (mostly security issues).

If you look to the past, various scrum teams tried dozens off tools.
But hey, you know what is still here? RedMine (and RHBZ of course :-)
- those systems survived. My sole opinion is let's ditch RedMine and
use RHBZ for everything, many open source projects do this. But I
understand many people cannot live with Bugzilla, that's fine. OK!
Let's just stick to RedMine then. If people want to try RedMine
plugins, I totally do support that. What I really like is to have
everything in one place - no copies.

Historically, there have been problems with RedMine plugins for scrum,
it was overloading the server when whole team was moving tickets. But
we might upgraded to better hosting how, if infrateam approves I am
all for trying anything that is a RedMine plugin and works with
regular issues.

This could be a chance to setup "staging" redmine, since we still have
some knowledge from the migration. There we can test it on our current
data and vote how we like it.

LZ

On Wed, Nov 29, 2017 at 8:00 AM, Ivan Necas  wrote:
> My uderstanding of kanban is, that moving the cards manually on the signal
> board is actually part of the kanban way of doing things. Another thing,
> that is integrated part of the kanban process is
> WIP limits, that I don't even see on the redmine plugin. Another reason for
> actually not using
> redmine for this is the rasks, that don't make sense to actually track in
> redmine.
>
> I have a pr to nice Marek's tooling around automating some common actions
> around
> Kanboard https://github.com/ares/kansync/pull/5 and so far, I'm happy with
> this.
>
> -- Ivan
>
> út 28. 11. 2017 v 22:37 odesílatel Andrew Kofink 
> napsal:
>>
>> I'm open to using other kanban style tools. To me it doesn't really matter
>> that much how we plan - it's the same information just in a different
>> format. So, if people have reasons, then I'm fine with switching.
>>
>> On Tue, Nov 28, 2017 at 4:29 PM, Walden Raines  wrote:
>>>
>>> Hey,
>>>
>>> With several teams moving to kanban style tools for the visualization of
>>> tasks I'm wondering if anyone has tried redmine's PluginKanban [1].
>>>
>>> I really like the idea of a trello/kanboard like tool but I hate having
>>> to update tasks in multiple places.  Another idea I had was to switch (back)
>>> to using GH issues and use GH projects [2] as well.
>>>
>>> Any other thoughts on how to make these kanban boards work better with
>>> redmine?
>>>
>>> Cheers,
>>> Walden
>>>
>>> [1] http://www.redmine.org/projects/redmine/wiki/PluginKanban
>>> [2] https://help.github.com/articles/about-project-boards/
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Andrew Kofink
>> akof...@redhat.com
>> IRC: akofink
>> Software Engineer
>> Red Hat Satellite
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman instrumentation for telemetry proposal

[Lukas Zapletal]

> 1. what happened to the PCP approach we talked about in the past?

Thats going in parallel, PCP is just a monitoring framework you can
integrate with instrumentation data just like any other.

> 2. how would you integrate this to sosreport/foreman-debug? I'm thinking of
> storing the statsd data locally, collecting them with foreman-debug, and
> then, being able to import them later to the prometheus and other tools. Is
> this how this could work? Any other options?

This is my ultimate goal to have working PCP deployment including
telemetry data and archives could be collected by foreman-debug, they
are pretty small (few MBs per day).

> 3. does every host/runtime needs it's own statsd service, or there would be
> one shared process? Asking bith for multi-host and containers use-case

It is up to you if you want one statsd service per guest/container,
host or subnet. Prometheus endpoint will not require any external
daemon once sharing metrics is merged into upstream. For this reason,
statsd will server as a temporary solution and alternative for the
future.

> The proposal of the telemetry api itself seems reasonable, let's discuss
> that on an actual PR

Thanks, I hope to finish it this year.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Discourse - next steps / actions

[Lukas Zapletal]

Although I prefer to only migrate users list, if we decide to migrate
both I would actually like to see foreman-announce to be migrated as
well. We previously had issues sending plugin announcements into this
low-volume list, it's a chance to change this and split it into
announce-core and announce-plugins or something like that. I assume
there would be an easy way of subscribing to those tags via RSS or
email.

LZ

On Thu, Nov 23, 2017 at 5:31 PM, Greg Sutcliffe  wrote:
> Hi all,
>
> I'll try to keep this brief, at least by my standards :)
>
> Firstly, I think it's clear that the discussion is leaning towards
> implementing Discourse for at least one, and possibly both, of the
> mailing lists. Many people have contributed - I'd like to thank them all
> for keeping this from descending into a flamewar :)
>
> We do need to give the users list a little longer for feedback (it's
> only been a week), but I think here we can start to talk about the
> migration planning.
>
> Since we all (or most?) need to start getting used to Discourse, I've
> written up a draft migration plan there:
>
> https://community.theforeman.org/t/draft-migration-plan-theoretical/7550
>
> I've just updated it to reflect the latest discussions. No doubt I have
> missed something though, so I've made it into a wiki post - feel free to
> edit with suggestions.
>
> There are two things I'd like to get input from people on:
>
> # Moderators
>
> I need a small team of moderators to deal with running the forum. I
> don't expect it to take up much time, but it is important that more
> people than just me know how things work. 3-4 people (including me) is
> probably enough to start with.
>
> So, volunteers needed! Don't make me start picking on people :)
>
> # Choice of lists to migrate
>
> As above, we have to decide if we migrate both lists or just one. I see
> significant benefits on having everyone on the same platform (top 2 -
> the @groups feature is very useful for discussions, and the
> discoverability of dev discussions for users helps them to get involved
> and grow our dev community). However, I know others are more cautious
> than I.
>
> Again, since we'll all likely be using Discourse in some capacity, I've
> gone ahead and created a poll on Discourse for us. Please do leave your
> feedback! Votes by email-response (here or on Discourse) are of course
> acceptable for those allergic to UIs, I will add them in ;)
>
> https://community.theforeman.org/t/poll-which-lists-should-migrate-to-discourse/7598
>
> I've not covered foreman-announce - we use this so infrequently, and
> posting rights are heavily restricted, so I think we can decide what to
> do with it later.
>
> Hang in there, we're getting to the fun bit! :)
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] 1.17 branching - status update

[Lukas Zapletal]

Thanks please announce the branching day a day ago and ask to stop
merging breaking changes, so we can test our plugins with the
"candidate" for branching. That would be super nice!

LZ

On Thu, Nov 23, 2017 at 2:03 PM, Ondrej Prazak  wrote:
> Hi,
> this is just a quick summary of where we stand with 1.17, please feel free
> to correct me if I got something wrong.
>
> foreman_selinux, smart-proxy, installer and modules have no blockers - at
> least none that we know of.
>
> Blockers for foreman are dynflow and Rails 5.1 as already stated previously
> [1]. I tried to get in touch with the person who opened a PR to fix
> turbolinks [2], but I got no response, so I opened a PR directly against
> turbolinks [3]. foreman-task needs to get rid of alias_method_chain [4] and
> katello needs to update accordingly [5].
>
>
> O.
>
> [1] https://groups.google.com/forum/#!topic/foreman-dev/3G9GuQ6d8ms
> [2] https://github.com/dobtco/turbolinks-classic/pull/1
> [3] https://github.com/turbolinks/turbolinks-classic/pull/679
> [4] https://github.com/theforeman/foreman-tasks/pull/300
> [5] https://github.com/Katello/katello/pull/7083
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Getting Foreman and Smart-Proxy to run in FIPS environment.

[Lukas Zapletal]

Thanks, so ABRT is raised when a FIPS-enabled library (e.g. openssl)
catches an attempt to use non-approved cipher.

Do you have some examples how such an abort looks like with and without GDB?

Why I am asking. Once we start supporting FIPS, I wonder how we are
gonna provide support for users and customers who are in FIPS mode and
"someting is breaking apart". Installing GDB would be really the last
thing to do. I wonder if there is any chance of using SystemTap or
even ABRT (Automatic Bug Reporting Tool) data (stacktrace with debug
info, core dump). Trying to find a workflow which we will use for
retrospective analysis to find the Ruby code which is causing this.

Was wondering if we can hook into ABRT and print stacktrace similarly
as we do for:

https://github.com/theforeman/foreman/blob/698e916ce208b5040b83a908a058c83c94d158ee/config/initializers/sig_ttin_trap.rb

LZ

On Mon, Nov 20, 2017 at 8:05 PM, Dmitri Dolguikh <witlessb...@gmail.com> wrote:
>
>
> On Mon, Nov 20, 2017 at 2:05 AM, Lukas Zapletal <l...@redhat.com> wrote:
>>
>> Perhaps - can you bit elaborate the GDB
>> thing? Is this some kind of hook that use used for FIPS stack to
>> report "mistakes" (e.g. signal or exception when you attempt to use
>> md5 hash)? I wonder if there is a way to catch these without GDB, for
>> example with SystemTap which would allow to catch these also in
>> production. Perhaps I misinterpret how this works, I haven't opened
>> the PDFs to be honest yet :-)
>
>
> DGB + ruby-specific .gdbinit are used to report ruby callstack when ABRT is
> caught. I didn't look for alternatives, seeing that gdb + .gdbinit
> combination is pretty trivial to use. I don't really care what we use here
> as long as we can get ruby callstack back.
>
>>
>>
>> I understand you only investigated core and smart proxy and I assume
>> plugin authors will need to do the similar audit themselves. Since we
>> are dealing with security, I would like to propose to create kind of
>> FIPS CHECKLIST somewhere (perhaps our wikipage) so we could follow it
>> and also add own findings/places to look for. In security world it's
>> much more useful to have blueprints or checklists, I would not like to
>> reinvent the wheel when doing this for discovery or other plugins.
>
>
> Sure, sounds like a good idea. I think it's going to consist of the list of
> hash-functions and ciphers not to use and a recommendation to test changes
> on a system with FIPS enabled.
>
>>
>>
>> Once we settle down on your proposal, please do create tracker
>> issue(s) so we can associate our audits or changes to it. If you find
>> anything that needs to be done for all plugins, please create tracker
>> issue as well.
>
>
> Tracker issue for Foreman already exists:
> http://projects.theforeman.org/issues/3511.
>
> -d
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman instrumentation for telemetry proposal

[Lukas Zapletal]

Thanks for question, this will be completely opt-in via
/etc/foreman/settings.yaml. You can turn off (default behavior when
not set), or on via prometheus, statsd or logging implementation (for
debugging purposes - sends stats to Rails log / production.log).

On Mon, Nov 20, 2017 at 4:31 PM, Bryan Kearney <bryan.kear...@gmail.com> wrote:
> How would folks disable it opt out of sending this data?
>
> On Nov 20, 2017 9:46 AM, "Lukas Zapletal" <l...@redhat.com> wrote:
>>
>> Hey,
>>
>> on the last demo I presented my proposal for telemetry (it is actually
>> a separate video). I am looking for non-intrusive approach with broad
>> integration possibilities:
>>
>> https://www.youtube.com/watch?v=gCLSI9-4QpE
>>
>> This was also showed on our demo last week (the same content):
>> https://www.youtube.com/watch?v=QHzNIFjMpTM
>>
>> I am starting this thread to gather feedback before I open a PR with
>> this. Currently the code is mostly in Rails initializer and looks like
>> this:
>>
>> # get telemetry singleton instance and setup it
>> telemetry = Foreman::Telemetry.instance.setup(... some options ...)
>>
>> # register measurements
>> telemetry.add_counter(:http_requests, 'A counter of HTTP requests
>> made', [:controller, :action])
>> telemetry.add_histogram(:http_request_total_duration, 'Total
>> duration', [:controller, :action])
>> telemetry.add_counter(:activerecord_instances, 'Number of instances of
>> AR models', [:class])
>>
>> # send measurements from Rails instrumentation or from code base
>> telemetry.increment_counter(:http_requests, 1, :controller =>
>> controller, :action => action, :status => status)
>> telemetry.observe_histogram(:http_request_total_duration, duration,
>> :controller => controller, :action => action)
>>
>> The proposed API is a single class (a singleton actually) with three
>> registering methods and three measure methods. I don't think such a
>> simple class needs proper separation of concerns, but we can talk
>> about this in the PR. The registration part could be turned into some
>> kind of DSL, currently it takes metric name, description and list of
>> keys which will be part of an instance for those frameworks which do
>> not support arbitrary amount of key-value pairs.
>>
>> If there are no objections, I will add settings and better error
>> handling and file the PR.
>>
>> --
>> Later,
>>   Lukas @lzap Zapletal
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Foreman instrumentation for telemetry proposal

[Lukas Zapletal]

Hey,

on the last demo I presented my proposal for telemetry (it is actually
a separate video). I am looking for non-intrusive approach with broad
integration possibilities:

https://www.youtube.com/watch?v=gCLSI9-4QpE

This was also showed on our demo last week (the same content):
https://www.youtube.com/watch?v=QHzNIFjMpTM

I am starting this thread to gather feedback before I open a PR with
this. Currently the code is mostly in Rails initializer and looks like
this:

# get telemetry singleton instance and setup it
telemetry = Foreman::Telemetry.instance.setup(... some options ...)

# register measurements
telemetry.add_counter(:http_requests, 'A counter of HTTP requests
made', [:controller, :action])
telemetry.add_histogram(:http_request_total_duration, 'Total
duration', [:controller, :action])
telemetry.add_counter(:activerecord_instances, 'Number of instances of
AR models', [:class])

# send measurements from Rails instrumentation or from code base
telemetry.increment_counter(:http_requests, 1, :controller =>
controller, :action => action, :status => status)
telemetry.observe_histogram(:http_request_total_duration, duration,
:controller => controller, :action => action)

The proposed API is a single class (a singleton actually) with three
registering methods and three measure methods. I don't think such a
simple class needs proper separation of concerns, but we can talk
about this in the PR. The registration part could be turned into some
kind of DSL, currently it takes metric name, description and list of
keys which will be part of an instance for those frameworks which do
not support arbitrary amount of key-value pairs.

If there are no objections, I will add settings and better error
handling and file the PR.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] CDN for the theforeman.org

[Lukas Zapletal]

Great, thanks. Before moving on, I'd check if yum/deb accepts these
kind of certificates. I *think* so, but I've heard from security guys
they don't like these certs at all :-)

LZ

On Fri, Nov 17, 2017 at 4:14 PM, Greg Sutcliffe  wrote:
> Hi all,
>
> I've signed the contact with our newest sponsor, Fastly, and we now have
> a CDN available for us to use on our web host. This comes with a single
> wildcard certificate, so we should be able to use it for all the vhosts
> on there (yum.tf, deb.tf, etc)
>
> Being entirely honest, I've never set up a CDN before though, so do we
> have anyone in the community who has?
>
> BTW, big thanks to Evgeni (Zhenech) for his help in getting this sorted!
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Koji repo sync metadata problem

[Lukas Zapletal]

Sorry we had public holiday here.

We use mirrorselect with mrepo, sometimes it can pick slower mirror.
Would be safer to pick particular mirrors if we plan to sync
regularly, ideally something within AWS datacenters if there's any.

LZ

On Fri, Nov 17, 2017 at 4:53 PM, Eric D Helms <ericdhe...@gmail.com> wrote:
> The update seems to have completed, with the new CentOS repository added to
> Koji and repositories regenerated. For the moment, the
> foreman-nightly*-rhel7-build tags have been updated to remove rhel as an
> external repository and have been replaced with CentOS (7.4). If you notice
> any oddities building, please let us know so we can work through them.
>
>
> Eric
>
> On Thu, Nov 16, 2017 at 7:14 PM, Eric D Helms <ericdhe...@gmail.com> wrote:
>>
>> I started an mrepo run after adding CentOS in a screen session. The run
>> appears to be running a lot longer than previous and it's not clear to me
>> what exactly it's doing at present or whats safe to do. Lukas could you
>> check on it in your morning and see if you spot anything odd about why it
>> seems to have stalled or slowed down?
>>
>> On Thu, Nov 16, 2017 at 6:01 AM, Ewoud Kohl van Wijngaarden
>> <ew...@kohlvanwijngaarden.nl> wrote:
>>>
>>> Those sound like good arguments, but if we don't test on older releases
>>> then we don't know for sure they work either. IMHO it's acceptable to
>>> explicitly only support EL 7.4 and up. We don't have the resources to
>>> support older versions especially since CentOS doesn't support that either.
>>> If users want verified working versions on EL < 7.4 then they should either
>>> help us properly support it or buy support from a vendor that does want to
>>> promise it works.
>>>
>>>
>>> On Thu, Nov 16, 2017 at 08:56:39AM +0100, Lukas Zapletal wrote:
>>>>
>>>> I would be against syncing regularly but I don't do much packaging so
>>>> it's up to you. The problem I see is "random" breakage. You come to
>>>> work on Monday after sync and if you don't realize the package was
>>>> deleted from EPEL, you can burn hour or something to figure it out
>>>> sometimes. This was pretty clear case but as you know there are
>>>> situations (SCL) when it is not that clear.
>>>>
>>>> If we ever deploy new koji, I'd vote to avoid getting OS updates
>>>> completely, just base channels and explicit updates when we need it.
>>>> We need reproducible builds, with autosync we are not getting it.
>>>>
>>>> One note, if we sync CentOS or RHEL base to particular version,
>>>> chances are that SELinux won't load on older releases. So basically we
>>>> must be ready to do minimum requirements change here.
>>>>
>>>> Thanks for solving the problem.
>>>>
>>>> LZ
>>>>
>>>> On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden
>>>> <ew...@kohlvanwijngaarden.nl> wrote:
>>>>>
>>>>> I think we should sync & update. CentOS has no concept of supported
>>>>> minor
>>>>> releases and we should be testing with a supported release.
>>>>>
>>>>>
>>>>> On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:
>>>>>>
>>>>>>
>>>>>> This now appears to be working again. One out standing question that
>>>>>> affects nodejs- packaging builds.
>>>>>>
>>>>>> As part of the RHEL 7.4 release, http-parser was removed from EPEL.
>>>>>> With
>>>>>> this latest round of Koji external repositories update we now have a
>>>>>> newer
>>>>>> EPEL with this package removed. We did not update our CentOS external
>>>>>> repository that would include this package. Not updating the base OS
>>>>>> has
>>>>>> been our policy for a while now it would seem. We have two options:
>>>>>>
>>>>>> 1) Sync and update CentOS
>>>>>> 2) Add http-parser to the overrides tag for foreman-nightly
>>>>>>
>>>>>> On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhe...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I had forgotten to run the repo-regen on Koji. That is finishing up
>>>>>>> now.
>>>>>>> I
>>>>>>> will run another test after and report back.
&g

Re: [foreman-dev] Koji repo sync metadata problem

[Lukas Zapletal]

I would be against syncing regularly but I don't do much packaging so
it's up to you. The problem I see is "random" breakage. You come to
work on Monday after sync and if you don't realize the package was
deleted from EPEL, you can burn hour or something to figure it out
sometimes. This was pretty clear case but as you know there are
situations (SCL) when it is not that clear.

If we ever deploy new koji, I'd vote to avoid getting OS updates
completely, just base channels and explicit updates when we need it.
We need reproducible builds, with autosync we are not getting it.

One note, if we sync CentOS or RHEL base to particular version,
chances are that SELinux won't load on older releases. So basically we
must be ready to do minimum requirements change here.

Thanks for solving the problem.

LZ

On Wed, Nov 15, 2017 at 10:05 PM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> I think we should sync & update. CentOS has no concept of supported minor
> releases and we should be testing with a supported release.
>
>
> On Wed, Nov 15, 2017 at 03:57:40PM -0500, Eric D Helms wrote:
>>
>> This now appears to be working again. One out standing question that
>> affects nodejs- packaging builds.
>>
>> As part of the RHEL 7.4 release, http-parser was removed from EPEL. With
>> this latest round of Koji external repositories update we now have a newer
>> EPEL with this package removed. We did not update our CentOS external
>> repository that would include this package. Not updating the base OS has
>> been our policy for a while now it would seem. We have two options:
>>
>> 1) Sync and update CentOS
>> 2) Add http-parser to the overrides tag for foreman-nightly
>>
>> On Wed, Nov 15, 2017 at 2:11 PM, Eric D Helms <ericdhe...@gmail.com>
>> wrote:
>>
>>> I had forgotten to run the repo-regen on Koji. That is finishing up now.
>>> I
>>> will run another test after and report back.
>>>
>>> On Wed, Nov 15, 2017 at 2:08 PM, Patrick Creech <pcre...@redhat.com>
>>> wrote:
>>>
>>>> On Wed, 2017-11-15 at 14:00 -0500, Patrick Creech wrote:
>>>> > On Wed, 2017-11-15 at 19:56 +0100, Lukas Zapletal wrote:
>>>> > > I do not have access here, can someone take a look? Are repodata
>>>> > > present for EPEL7?
>>>> >
>>>> > If I have the correct path, it does not appear so:
>>>> >
>>>> > [root@koji repodata]# pwd
>>>> > /mnt/koji/external-repos/epel7-x86_64/epel/repodata
>>>> > [root@koji repodata]# ls
>>>> > [root@koji repodata]#
>>>> >
>>>>
>>>> I might not have, this path DOES have repodata:
>>>>
>>>> /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/repodata
>>>>
>>>> > > LZ
>>>> > >
>>>> > > On Wed, Nov 15, 2017 at 6:15 PM, Ewoud Kohl van Wijngaarden
>>>> > > <ew...@kohlvanwijngaarden.nl> wrote:
>>>> > > > I still get errors when building:
>>>> > > >
>>>> > > > http://koji.katello.org/koji/taskinfo?taskID=52363
>>>> > > > http://koji.katello.org/kojifiles/work/tasks/2363/52363/root.log
>>>> > > >
>>>> > > > DEBUG util.py:439:  Error downloading packages:
>>>> > > > DEBUG util.py:439:1:nodejs-6.10.3-1.el7.x86_64: failed to
>>>> retrieve
>>>> > > > nodejs-6.10.3-1.el7.x86_64.rpm from build
>>>> > > > DEBUG util.py:439:  error was [Errno 2] Local file does not exist:
>>>> > > > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/nodejs-6
>>>> .10.3-1.el7.x86_64.rpm
>>>> > > > DEBUG util.py:439:http-parser-2.7.1-3.el7.x86_64: failed to
>>>> retrieve
>>>> > > > http-parser-2.7.1-3.el7.x86_64.rpm from build
>>>> > > > DEBUG util.py:439:  error was [Errno 2] Local file does not exist:
>>>> > > > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/http-par
>>>> ser-2.7.1-3.el7.x86_64.rpm
>>>> > > > DEBUG util.py:439:1:npm-3.10.10-1.6.10.3.1.el7.x86_64: failed
>>>> to
>>>> > > > retrieve npm-3.10.10-1.6.10.3.1.el7.x86_64.rpm from build
>>>> > > > DEBUG util.py:439:  error was [Errno 2] Local file does not exist:
>>>> > > > /mnt/koji/external-repos/www/epel7-x86_64/RPMS.epel/npm-3.10
>>>> .10-1.6.10.3.1.el7.x86_64.rpm
>>>> > > >

Re: [foreman-dev] Request to add Fedora 27 to Koji

[Lukas Zapletal]

We have 200GB spare space, one more Fedora could fit. Can we delete
Fedoras 24 and 25? This is definitely the last time before we'd need
to expand space again.

I initiated mrepo dry run which turned into real run and koji is now
out of service due to missing metadata, we are working on this. Will
get back to you, will take several hours.

On Wed, Nov 15, 2017 at 1:59 AM, Patrick Creech  wrote:
> Hi,
>
> With the release of Fedora 27 today, pulp has a need to add Fedora 27 support 
> for pulp's upcoming
> 2.15.0 release, if possible.  While the transition to copr is on the table 
> for pulp, we don't
> anticipate completion before 2.15.0 goes out.
>
> With that, is it possible to get Fedora 27 repos added to koji if space 
> allows?
>
> Thanks,
> Parick
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Koji repo sync metadata problem

[Lukas Zapletal]

Hey,

when I initiated mrepo -n (dry run) this morning to see how mrepo
works on our koji in order to test if we are able to add Fedora 27 for
Pulp, I learned that this "dry run" is actually a real run and mrepo
initiated full resync of our content without metadata regeneration.

This rendered our external repositories unusable as all metadata were
deleted. All builds will fail today with missing dependencies.

The plan: we need to let mrepo finish "dry run" downloading of Fedora
27 and then we need to reexecute it with proper flags:

cat /usr/local/bin/koji-sync-external-repos
#!/bin/bash
mrepo -qugf
koji list-targets --quiet | awk '{print $2}' | sort -u | xargs -I '{}'
koji regen-repo --nowait '{}'

Note the -u -g -f flags for mrepo, this will cause all metadata to be
regenerated. Once this command is done, koji will see missing
dependencies to appear.

The dry run of mrepo is running in screen and it is about to finish
hopefully soon. I will defer this to Eric in case it won't finish by
EMEA EOB.

Sorry for inconvenience, but dry run should be dry run, right.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] [RFC] Deprecate and move Github RFC repo content

[Lukas Zapletal]

Hey,

this is another proposal to deprecate Github RFC repo and move the
content to our wiki. Reasons:

A) There is zero merged proposals for the past year (Jan-Nov 2017):
https://github.com/theforeman/rfcs/commits/master

B) Activity is very low (comments in 3 PRs last winter, 3 PRs last
summer, 1 PR last month):
https://github.com/theforeman/rfcs/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc

C) There is only one published RFC on the front page (by the way I am
the author). I see some randomly numbered files in text/ folder but
this does not look like a friendly space to newcomers or even
experienced guys. We failed following processes described on the front
page obviously and it's no surprise - people in need of writing a
proposal need to focus on the most important stuff.

D) Developers naturally create [RFC] threads on our -dev list, some
random subjects:
* Make foreman STI to work even when an inherited class is not found
* Proxies with multiple interfaces
* Found In Version Katello Redmine custom field
* Foreman with Puppet in a wildcard domain leads to nodes mistaken identity
* Templates rendering only through Proxies
... and others I am gonna stop here. These are e-mail from core devs
and also our community.

E) It's been already proposed on list by Tomer ("Retire the RFC
repo"). Some people expressed intentions to use it (Justin, Eric, John
and Perry) but I haven't seen much activity (see above). One of the
reasons was time, but these days we should be pushing hard on
proposals for the next generation of Foreman versions and I don't see
this. Greg agreed this needs some more work around visibility and
process improvement, but I don't see much improvements.

F) Content is not visible, most of us don't use it and WIP RFCs are
quite unreadable. I want to read proposal as a completed document and
then I want to send a consistent opinion about it.

PROPOSAL

Let's close the RFC repo for new PRs, add a note that contents has
been moved. I will then move all finished RFCs to our wiki page. We
already have that for years, it's named Features:

http://projects.theforeman.org/projects/foreman/wiki/Features

There is a template

http://projects.theforeman.org/projects/foreman/wiki/Sprint_Feature_Template

And couple of proposals already:

http://projects.theforeman.org/projects/foreman/wiki/Pagelets
http://projects.theforeman.org/projects/foreman/wiki/Discovered_host_redesign
http://projects.theforeman.org/projects/foreman/wiki/Remote_Execution_Design
http://projects.theforeman.org/projects/foreman/wiki/CentralizedLogging
http://projects.theforeman.org/projects/foreman/wiki/PXE_Booting_UEFI
(I moved this one, I take this back)

The wiki pages are just a place to put content on, a opt-in way. Most
of us will likely prefer e-mail list for smaller proposals and I
encourage everybody to continue to do so. This is just a complementary
place, but it's not pulling conversation out of our devel list.

BENEFITS

1) Stops splitting place for important discussions - we already have
the dev list.
2) Actually *readable* content, easy to edit, easy to track changes.
3) Rich content - github hasn't invented HTML/RTF with images, wiki
can do it all.
4) Free form - just write your proposal, add a title, name and status
and that's it.

ALTERNATIVE PROPOSAL

Let's keep the github repo but remove all git content and move all
pages to wiki section under that github repository. Discussion will be
made in the devel mailing list. Advantage is that the syntax will
remain the same, but I hereby offer migration of all finished RFCs
into RedMine syntax.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

I will just reply to phased rollout.

> Also, this may be wishful, but the fact that
> people choose to post on StackOverflow rather than our mailing list
> again shows a desire for more modern forms of interaction.

Why are we creating a duplicate of StackOverflow then? There is a
community already, let's just embrace it, advertise on dev channels,
create links from site, blog post and integrate more tightly with
-user list. (*)

I don't really see much value in building another StackOverflow. You
are not sure that these people which are comfortable with discussing
there will move to Discourse.

(*) https://stackoverflow.blog/2010/12/20/subscribe-to-tags-via-emai/

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: Foreman instrumenting analysis

[Lukas Zapletal]

Thanks for tip, I considered InfluxDB but I have major problem with
that. It's a backend system rather than open protocol with many
implementations, more of a database rather than monitoring protocol.

But it is a viable option, the it has a push API which is HTTP JSON
based, which will be much slower than statsd (just a UDP packet).

Let me see if we settle down on my proposal which I just sent of
having two implementations in Foreman core - one push, one pull. I can
even do just common API in Foreman core and write the two
implementations as plugins, then adding a new one (InfluxDB) would be
quite easy if needed.

On Tue, Nov 7, 2017 at 3:05 PM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> Have you had a look at influxdb? While I have limited experience with it,
> it's push based which can have benefits. There's infludb-rails[1] which
> states:
>
> Out of the box, you'll automatically get reporting of your controller, view,
> and db runtimes for each request.
>
> That sounds a lot like what you're interested in.
>
> [1]: https://github.com/influxdata/influxdb-rails
>
>
> On Tue, Nov 07, 2017 at 10:49:56AM +0100, Lukas Zapletal wrote:
>>
>> Any other ideas for telemetry protocols?
>>
>> If there are none, I will rebase my telemetry patch back to the
>> original version based on statsd.
>>
>> LZ
>>
>> On Tue, Oct 31, 2017 at 8:33 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>
>>> Hello,
>>>
>>> I am seeking for app instrumenting protocol for Foreman Rails
>>> application that will fulfill the following requirements:
>>>
>>> The protocol must work with multi-process server like Passneger.
>>> The protocol can be easily integrated into Foreman Tasks and Smart Proxy.
>>> The protocol or agent must support aggregation of time-based data
>>> (quantiles, average).
>>> The protocol must integrate with top three open-source monitoring
>>> frameworks.
>>>
>>> Let me summarize my findings so far. I am looking for advice or
>>> comments on this topic. I already worked on some prototypes, but
>>> before I commit to some final solution, I want to be sure I will not
>>> miss something I don't know about.
>>>
>>> Before you send comments, please keep in mind I am not searching for
>>> monitoring solution to integrate with. I want an application
>>> instrumentation library (or protocol) to be able export measurements
>>> (or telemetry data if you like) from Rails (like number or requests
>>> processed, SQL queries, time spent in db or view, time spent rendering
>>> a template or calling a backend system).
>>>
>>>
>>> Prometheus
>>>
>>>
>>> Flexible text-based protocol (alternatively protobuf) with HTTP
>>> REST-like communication. It was designed to be pull-based, meaning
>>> that an agent makes HTTP calls to web application which holds all
>>> metrics until they are flushed. It was build for Prometheus monitoring
>>> framework (Apache licenced) created by SoundCloud initially. Server
>>> and most agents are written in Go, can run without external database
>>> or export into 3rd party storage backends.
>>>
>>>
>>> It looks great, but it has a major problem - the Ruby client library
>>> (called client_ruby) does not support multi-process web servers at
>>> all. There are some hacks but these are using local temp files or
>>> shared memory with rather bad benchmark results (see the links down
>>> below).
>>>
>>>
>>> There is a possibility to push metrics into a separate component
>>> called PushGateway, but this was created for things like cron jobs or
>>> rake tasks. Doing multiple HTTP requests for each metric per single
>>> app request will unlikely perform well. In the README authors have
>>> note that this should be considered as "temporary solution".
>>>
>>>
>>> Although Prometheus seems to have vibrant community, the Ruby library
>>> development pace slowed down as SoundCloud "does not use many Ruby
>>> apps anymore". But it is still a good option to have.
>>>
>>>
>>> https://prometheus.io
>>> https://prometheus.io/docs/instrumenting/pushing/
>>> https://github.com/prometheus/client_ruby
>>> https://github.com/prometheus/client_ruby/issues/9
>>> https://github.com/prometheus/client_ruby/commits/multiprocess
>>>
>>>
>>> OpenTSDB
>>>
>>>
>>> OpenTSDB consists of a Tim

Re: [foreman-dev] Foreman instrumenting analysis

[Lukas Zapletal]

> In reading about push vs. pull, my biggest issue with it is that the
> application has to have knowledge of where it's pushing. Whereas the pull
> model allows an application to say I have metrics here and anything that
> knows how to scrape and interpret those metrics can grab them at their
> leisure. This provides nicer de-coupling and potentially more choice if
> there is a standard-ish data format used to expose the metrics.

On the other hand, pull model requires to store the data in memory or
disk. That's more code in our app, wasting cpu cycles and memory,
because amount of telemetry data can be large and the only reasonable
way of handling that (that's what telemetry clients pull libraries do)
is aggregating data (more cpu cycles spent) and bigger chance for
errors. I personally see this as bigger issue than knowledge where to
push data. In statsd for example, it's standard practice to push to
localhost which is sane default.

But I understand your concerns let me elaborate my proposal on a
common API below.

> The biggest limiting factor appears to be how forking webservers are handled
> and probably constraints us the most. Lukas, have you seen anything related
> to being to define what the metrics are and how they get published being
> able to be separated from the publishing mechanism? My thinking being if we
> started with statsd and wrote code within the application generating statsd
> metrics, if at a later point one could simply say now publish this via HTTP
> endpoint in Prometheus data style for scraping?

Instrumenting in both statsd and prometheus are very similar
interfaces and writing a simple API that would allow us to switch over
to Prometheus and Statsd would be pretty easy, I can incorporate this
into my patch and even provide Prometheus support from the day one, we
would have two dependencies: statsd and prometheus ruby libraries, not
big deal. We just need to do big warning that it won't work in
multiprocess deployment so users won't accidentally enable this losing
lot of data. Let me describe both APIs:

Prometheus (https://github.com/prometheus/client_ruby)

Types: Counter, Gauge, Histogram (time data), Summary (we would not
use this as it is not recommended and CPU heavy)
Metric naming: single string with underscores (my_metric_name)
Tag support: Any metric can have key/value pairs, arbitrary amount.
Previously there was "instance" flag, but that is now handled via tag
named "instance".

Statsd (https://github.com/etsy/statsd/blob/master/docs/metric_types.md)

Types: Counter, Gauge, Time
Metric naming: single string with dots (my.metric.name)
Tag support: Does not support by default, but can be mapped into name.
There is a common protocol extension tho, any metric can have
key/value pairs as well, I would not use it for now.

Instances are important, allows us to group common metric, e.g.
request_processed would be counter of overall requests with instances
of all HTTP codes (200, 201, 500 etc) so you can easily compare those
on a single graph. On the other hand arbitrary tags are not that
useful, I would not add them in the first phase although there are
common statsd extensions to support it.

PROPOSAL FOR COMMON PROMETHEUS/STATSD API

Types: Counter, Gauge, Time (Histogram in Prometheus, Time in Statsd)
Metric naming: Single string with underscores, Prometheus naming
conventions (https://prometheus.io/docs/practices/naming/)
Instance name: Single string with underscore (or can be nil if there
is no instance for a metric)

For Prometheus this is natural mapping, no surprises there. For statsd
which is little bit more free form I propose this mapping:

metric_name.optional_instance_name

E.g.:

http_request_duration_seconds.architectures_show

Using this simple approach we can have a global setting that user can
change - no telemetry, telemetry via rails logger, prometheus and
statsd.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] speeding up tests

[Lukas Zapletal]

Nicely written.

> Ruby 2.2 -- postgresql: 48 minutes
> Ruby 2.2 -- mysql: 30 minutes
> Ruby 2.2 -- sqlite3: 26 minutes
> Ruby 2.3 -- postgresql: 56 minutes
> Ruby 2.4 -- postgresql: 57 minutes

Fun fact, the newer Ruby version the slower time.

> Setup: 12 minutes

This is one major pain we have we need to solve.

> Let's remove tests not
> because they don't break, but because they aren't providing any real value.

That's what I tried to articulate, not my idea at all. I've seen some Bob talks.

> Let's also focus on where the bottlenecks lie. If we remove 20 unit tests
> because they are old and don't break, but run in 100 ms each, we've only
> shaved 2 seconds off of a 45 minute test run. Jenkins output for a given PR
> gives a view of what the slowest test suites are. I'd encourage developers
> to start there and tackle the bottlenecks [3].

Yeah very relevant point.

> 2) Re-examine Parallel of Testing

For foreman-core tests with discovery, I am able to pull this down to
less than 4 minutes on 8 core CPU, but I see about 60 failures mostly
JavaScript. Most of these errors are:

TypeError: undefined is not an object (evaluating 'tfm.nav.init')

If you want to give this a try, it is as easy as:

be rake parallel:create
be rake parallel:migrate
be rake parallel:test

For more details visit:
https://github.com/theforeman/foreman/pull/2299

> 3) "Static" Environments
>
> This comes from both a trend in the Jenkins world and a question from Timo
> to move towards essentially Docker containers with pre-populated
> environments. The idea would be to ensure these are routinely updated with
> baseline dependencies installed so that setup time is reduced by having most
> assets (e.g. gems and npm modules) already on the container.

I don't think we necessary need containers for that, I think putting
those modules and gems into git (e.g. via git annex) or to some
side-repo could do the trick. Also migrations could be solved using
similar technique.

I think introducing containers can be counter-productive. You need to
build those containers in the end, that costs time and resources.

> 4) Quick to Fail
>
> This is the idea that we have the test suite bail out on the first failure,
> freeing up Jenkins and giving feedback sooner that the PR has issues. I am
> not a huge fan of this as it has some downsides for developers who rely on
> Jenkins to tell them what issues their work has versus running tests locally
> first.

I actually like this. Although I also sometimes send small patch
without running tests and rely on Jenkins, I have no problems running
them locally when it blows up.

> 6) Reduce Support Matrix

Please let's do this. I think that maximum two Ruby versions is enough
(oldest and newest supported) and we could skip MySQL for PR tests
(but keep it in full test suite in master/develop branches).


-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] speeding up tests

[Lukas Zapletal]

> We can define these tests as a seperate set that gets only a weekly run,
> but deletion is IMHO the completetly wrong way.

https://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

A cleanup *is* reasonable, not completely wrong way. I bet that week
after we move this into 2nd tier it starts breaking and nobody will
care, because code would had been merged already. That's the point
here.

> I think both are needed. Integration tests are much slower than unit
> tests, in general.

And by the way our unit tests are not fast, really. They are slow as
hell, mainly because most of them are not really unit tests. I am
speaking to you, model tests. Let me give you an example:

test/models/architecture_test.rb

Not a unit test, not an integration test, not a system test, haven't
failed for a very long time, useless if we have the most important
stuff covered in an integration or system test, which we do.

--
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Re: Foreman instrumenting analysis

[Lukas Zapletal]

Any other ideas for telemetry protocols?

If there are none, I will rebase my telemetry patch back to the
original version based on statsd.

LZ

On Tue, Oct 31, 2017 at 8:33 PM, Lukas Zapletal <l...@redhat.com> wrote:
> Hello,
>
> I am seeking for app instrumenting protocol for Foreman Rails
> application that will fulfill the following requirements:
>
> The protocol must work with multi-process server like Passneger.
> The protocol can be easily integrated into Foreman Tasks and Smart Proxy.
> The protocol or agent must support aggregation of time-based data
> (quantiles, average).
> The protocol must integrate with top three open-source monitoring frameworks.
>
> Let me summarize my findings so far. I am looking for advice or
> comments on this topic. I already worked on some prototypes, but
> before I commit to some final solution, I want to be sure I will not
> miss something I don't know about.
>
> Before you send comments, please keep in mind I am not searching for
> monitoring solution to integrate with. I want an application
> instrumentation library (or protocol) to be able export measurements
> (or telemetry data if you like) from Rails (like number or requests
> processed, SQL queries, time spent in db or view, time spent rendering
> a template or calling a backend system).
>
>
> Prometheus
>
>
> Flexible text-based protocol (alternatively protobuf) with HTTP
> REST-like communication. It was designed to be pull-based, meaning
> that an agent makes HTTP calls to web application which holds all
> metrics until they are flushed. It was build for Prometheus monitoring
> framework (Apache licenced) created by SoundCloud initially. Server
> and most agents are written in Go, can run without external database
> or export into 3rd party storage backends.
>
>
> It looks great, but it has a major problem - the Ruby client library
> (called client_ruby) does not support multi-process web servers at
> all. There are some hacks but these are using local temp files or
> shared memory with rather bad benchmark results (see the links down
> below).
>
>
> There is a possibility to push metrics into a separate component
> called PushGateway, but this was created for things like cron jobs or
> rake tasks. Doing multiple HTTP requests for each metric per single
> app request will unlikely perform well. In the README authors have
> note that this should be considered as "temporary solution".
>
>
> Although Prometheus seems to have vibrant community, the Ruby library
> development pace slowed down as SoundCloud "does not use many Ruby
> apps anymore". But it is still a good option to have.
>
>
> https://prometheus.io
> https://prometheus.io/docs/instrumenting/pushing/
> https://github.com/prometheus/client_ruby
> https://github.com/prometheus/client_ruby/issues/9
> https://github.com/prometheus/client_ruby/commits/multiprocess
>
>
> OpenTSDB
>
>
> OpenTSDB consists of a Time Series Daemon (TSD) as well as set of
> command line utilities. Interaction with OpenTSDB is primarily
> achieved by running one or more of the TSDs. Each TSD is independent.
> There is no master, no shared state so you can run as many TSDs as
> required to handle any load you throw at it. Each TSD uses the open
> source database Hadoop/HBase or hosted Google Bigtable service to
> store and retrieve time-series data.
>
>
> It uses push mechanism via REST JSON API with alternative
> "telnet-like" text endpoint. Although it does have some agents, it is
> more used as a storage backend than end-to-end monitoring solution.
>
>
> http://opentsdb.net/overview.html
>
>
> Statsd
>
>
> Main idea behind this instrumentation protocol is simple - get the
> measurement out of the application as fast as possible using UDP
> datagram. A collector agent usually runs locally, it does aggregation
> and relays the measurements to target backend system. The vanilla
> version does not support tagging, but there are extensions or mappings
> possible to support that.
>
>
> Almost all monitoring platforms has some kind of
> agent/importer/exporter that talks via statsd. The original statsd
> daemon was written in Perl years ago, then it was re-popularized by
> node.js implementation, but there are many alternative agents from
> which the most promising is statsite with very easy extensibility.
>
>
> This protocol is my favourite because it plays well with multiprocess
> Ruby servers or other Foreman components (all can just send UDP
> packets to localhost) and it also takes all aggregation and storing
> temporary data out of Ruby application. It also brings chances of
> regressions in our codebase to bare minimum 

Re: Draft v2 - Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

> No, I don't think we can. We've been over this in the preceeding emails.
> We can give an archive a shiny GUI, but fundamentally they have a
> different feature set, and a different target user group. Your position
> on Discourse is clear, and recorded here for all to see. If enough
> people agree with you, it won't happen.

HyperKitty is not just a nice archive, you can easily just use mailing
list as a forum. There are other options like Gmane and others, but I
find HyperKitty as the nicest and smoothest experience. It is *the*
right interface into mailing list, which some of us want to continue
using. I just want to repeat it here in the new thread for others.

It's not just in hating Discourse, I particularly do not like big bang
migration. I would not write a single word if you'd propose installing
Discourse next to our existing communication channels. We have a lot
of people who subscribed for something and you are forcing them onto
something they haven't subscribed for. And we won't read their opinion
here until you do the migration and they will see (the terrible - I
think) experience. I'd prefer slower-paced rollout with old services
fading out (might not happen), therefore I'd like to propose other
options:

1) Ditch Discourse and move to Fedora Hosted mailinglists with
HyperKitty interface (*)
2) Phased rollout: Install Discourse and keep our -user and -dev lists
on Google Groups
3) (2) + integrating Discourse with -user list in some way (daily
summary mail to -user list)
4) Migrate only user list and keep -dev list on Google Groups

(*) Yeah still my preferred way, you can easily do new posts *without
subscribing* or doing *any* e-mail:

https://lists.fedoraproject.org/archives/list/test-mailm...@lists.fedoraproject.org/message/new

--
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] speeding up tests

[Lukas Zapletal]

We should start tracking test breakages in time, if we have a test
that did not break for a year, lets delete it. As simple as that. It's
added value is almost zero.

This is not out of my head, I've heard that on a talk somewhere.
Instead of deleting, we can move it to archive - run them on a nightly
basis, if we find this too much. Frankly, I prefer deletion.

Also, let's ditch unit tests in favor of integration tests. We have a
lot of areas covered with both and this is extra work and slowing down
things.

LZ

On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy  wrote:
> Hi,
>
> The challenge:
>
> As a developer, I feel under utilized waiting for tests, the current test
> suite for foreman core takes over an hour, multiply it by number of really
> simple code fixes based on comment and it takes up a good chunk of a day.
>
> I would like to suggest a short and long term solution, and would appreciate
> your feedback.
>
> Short-term:
>
> Break down test matrix into more specific groups, this will allow to
> re-trigger just a smaller part of the tests vs. the entire suit today, for
> example:
> * API tests
> * DB migrations
> * UI Integration tests
> * ...
>
> Long term:
>
> Break down core into smaller repositories - This is probably a bigger topic
> then just tests, but its worth raising in this context as well.. the
> benefits i see are:
>
> * smaller repos - faster tests per repo (we would still need a plugin matrix
> kind of tests done at some point)
> * better review process - smaller repos would mean the maintainers of the
> repo actually care for all/most of the pr's - today its not the case - many
> PR's are handled by sub groups (e.g. anything under webpack folder is
> "someone else")
> * potentially better API between parts - e.g. we can create a schema
> specific repo (not saying its a good idea) - which will always work with a
> dummy rails app - in the long run - this will ensure our schema is resilient
> to upgrades and model changes in core.
>
> I would even go further and enable auto merge after review + successful
> tests, e.g.
> 1. PR is sent
> 2. repo tests are executed and pass
> 3. reviewer approve the change
> 4. larger test matrix is executed and pass
> 5. code get auto merged.
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

Greg, I absolutely understand the motivation, every two years amount
of programmers doubles. That is a crazy amount of newcomers. But these
new people are not idiots and some technical level is required even
for soft roles in our community. And we can make lists approachable
very much like forums.

Do not put me into position of blind and angry dev who can't accept
something different or new. I understand all contexts and I say
Discourse is an overkill that will bother me and possibly others. God
I wish Google Groups are gone, but not for this.

> * do nothing

Honestly, yeah.

> * switch mailing list for minimal improvement

s/minimal/reasonable/

> * switch to a forum, big upheaval but potential big payoff

Sure, because there are no downsides.

It's not about a list standard e-mail headers. The forum has different
workflow and features and there will be new features as well while
mailing list will stay the same. This will screw my inbox. This will
but a wall between e-mail users and web forum users. This is what's
this all about. And I think we don't need to go that direction.

LZ

On Fri, Nov 3, 2017 at 6:45 PM, Greg Sutcliffe  wrote:
> One more thought occurred to me while I was out on the nursery pickup, so 
> I'll drop here before I bow out for the weekend.
>
> Lukas, I think part of our disagreement is our different goals. As I 
> highlighted in the last mail, users behave differently to devs. These days I 
> consider myself more user than dev (when did I last contribute code), so I 
> have a different world view.
>
> You want to protect a tried and trusted workflow, likely used by many here - 
> that's fine. My job is to promote and develop the user community, so I see 
> room for improvement.
>
> Here's the catch though... Our future devs, as a community, *come from* the 
> user community. If we don't focus there, then we risk stagnating the dev 
> community too.
>
> I won't deny this change is a larger net benefit for the user group. The case 
> for the dev community is harder to argue. But there *is* benefit, and 
> compared to running a list (for dev) and a forum (for users) I think the 
> better argument is to use a forum for both.
>
> I don't expect to convince everyone, so this is going to come down to a group 
> decision - but not for a while yet. We need to do more tests.
>
> Have a great weekend all,
> Greg
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Rails log is now printed for failed tests

[Lukas Zapletal]

Hey,

just to want to increase visiblity of this patch:

https://github.com/theforeman/foreman/pull/4739

In short, when you have a test failure, there should be a relevant
Rails log printed on standard output. You can opt-out this via ENV
variable if you don't like this, but I find this very useful
particularly on our Jenkins when debugging tests.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

> So if I understand, you're OK with a web interface for creating /
> managing a conversation that you don't want in your inbox in Hyperkitty,
> but not happy with exactly the same workflow in Discourse? I find that
> hard to resolve, can you clarify?

All I want is standard mailman-like list experience. This is a
regression in behavior, a change from a list to forum. Let me explain
down below.

> I disagree with the word "hack" here, I think Discourse also works fine
> over email. In all the testing I've done, the only thing I'm seeing is a
> minor issue with format=flowed URLs (and tbh not all mail clients
> support that either). You can set the mailing-list mode flag and it will
> email you everything, you need never log into the UI again. I've had
> that flag set for the last week, and can confirm that I was recieving
> everything and could reply/start threads just fine. How is this
> different to a mailing list?

I understand how this is supposed to work technically, but will my
gmail.com handle this correctly? This is not how e-mail lists are
supposed to work. Will my MUA work correctly? Won't I see broken
threads because folks will introduce new shiny feature into Discourse
that does not play with plain emails anymore? I can't tell. Yeah you
set up a testing instance, but there is almost no real traffic and I
really don't know how it will look like with thousands of emails. My
MUA handles millions of them just fine.

Mailman will be here for another 20 years and more. It will work as
long as e-mails are alive. And I do believe it's not going anywhere.

> It has to be said, I do like Hyperkitty, and I agree it would be an
> improvement over Google Groups. However, it doesn't address many of the
> use cases in my original post (and followups):

All of these are nice things, Greg. But these are irrelevant.

What we are running is a mailing list here. It's a mailing list, we
are open source developers and this is our way of communication.

People do not need "powers". Powers to do what with? Lock topics?
Create complicated sub-groups structures? Mute people or delete posts?
This is what I hate about forums. Pins and banners? Totally hate
these. Ever been to XdaDevelopers searching for Android ROM? God
that's awful, this is how it all ends - complicated pinned neverending
threads with things like "This post is reserved". And dozen of
"recommended topics" which are always totally irrelevant.

Let's just run a mailing list please. If you want to do your
statistics or gamify this, take this offline. There are tools
generating mail statistics. Fedora have concept of badges, it is all
doable.

> The summary seems to be that Discourse can do *everything* Hyperkitty
> can, and from my testing, even the email support is first class. Both
> are open, both can be hacked on if needed, both can be hosted or
> self-hosted. But only Discourse brings more power and flexibility to the
> table, both immediately, and for anything other needs we may have in
> the future.

You are comparing apple and orange here. A forum with a mailing list.
I say let's stick with an orange, a mailing list.

My response is not about "look here is an alternative HyperKitty". I'd
be fine with any other mailing list (*) with nice user archive with
easy entry for newcomers. It's about keeping our discussion where it
belongs.

(*) mailing list - a software that keeps emails in plain form not in
some kind of a database

> I'm *very* interested to hear others views though!

Yeah, feels like I am the only one. Not a good feeling, really.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

I was planning not to reply, but I can't sorry.

>>> I don't like Discourse at all. <<<

I was hoping to get rid of Google Groups in favour of - wait for it -
GNU Mailman! Yeah, you might think this would be step backwards, but
let me explain.

GNU Mailman is the core of open source. It's the heart, really.
Similarly as git or other things. It helps to communicate millions of
people for decades and it is nothing wrong with it. What is
problematic and old is the entry barrier (it's a list) and the default
mailman archival tool you may have already seen. This is how it looks
like:

https://lists.gnu.org/archive/html/weechat-dev/

Yeah, old, clunky interface and there is a need for new users to
register into the list to be able to post. But hey, this has been
already solved! It's called HyperKitty and Fedora already migrated to
a year ago! Look, nice and clean interface with ability to send
message right away from your browser:

https://lists.fedoraproject.org/archives/

When you click on Sign In, there are about a dozen of account
integrations including Google Plus, OpenID, GitLab, GitHub, Twitter,
Facebook or StackExchange. That's plenty, way enough for our users to
get started.

Then click on an archive (not list) and click on Start New Thread.
Yes, as easy as that. You can test on this testing mailing list:

https://lists.fedoraproject.org/archives/list/test-mailman3%40lists.fedoraproject.org/

I sometimes use this for lists I don't want to get into my inbox on a
daily basis and it works just fine. To start a thread, it's just a
simple link and HyperKitty will just ask you to Sign-In with one of
your existing accounts. Super fast:

https://lists.fedoraproject.org/archives/list/test-mailm...@lists.fedoraproject.org/message/new

Benefits? Mailman works great over email, it's not just some hack, it
its main purpose. You also have a lot of options when you are
receiving mail (daily, weekly, no copies etc), data is yours and not
somewhere in cloud if you want to, archives are very fast and
searchable via Google or anything pretty much. And your whole history
is just bunch of emails, data liberation in practice. When I compare
this to Google Groups or Discourse it also loads faster as there is
less JavaScript I think.

>>> My proposal is: Let's move to Fedora mailing lists with HyperKitty, it's a 
>>> reliable infrastructure. Problem solved. <<<

I am sorry Greg, I know you invested a lot in this and you have some
positive experience in the past. Well my experience with Discourse is
rather bad, mostly from https://forum.turris.cz/ but if there is some
chance to fix our terrible Google Groups, let's do it the "right way".
I also understand you might not be fully available for couple of next
months and it's just you if I understand correctly with experience
maintaining Discourse. I would rather preferred hosted solution on a
free as in freedom service (Fedora lists).

On Wed, Nov 1, 2017 at 4:54 PM, Greg Sutcliffe  wrote:
> Hi all,
>
> As ever from me, this is long. Sorry about that, it's a habit. Here's
> the TL;DR:
>
> * What: move Google Groups -> Discourse
> * Why: https://blog.discourse.org/category/use-cases/
> * Can I try? - Scroll to the end for login details
>
> So, as some of you know, I'm a big fan of the Discourse[1] forum
> software - I use it for another community, and it's just lovely. I've
> been testing it out recently with a view to using it for Foreman, and
> I think it's time to explain my reasoning and ask for your thoughts.
>
> # What?
>
> Firstly, the "what" - what do I want to do? Simply put, I want to
> migrate from Google Groups to Discourse. That means locking the groups
> from further emails and using Discourse for our "written" discussions.
> Obviosuly there's data migration that needs to happen there, but we'll
> get to that.
>
> Before all the die-hard mailing list fans stop reading here - please
> keep reading. Discourse has options to interact entirely by email.
> Your workflow may not be broken :)
>
> # Why?
>
> Why do I want this? The short version is "because anything is better
> than Google Groups", but more seriously, I think Discourse is great.
> The reasons are different for each of our mailing lists though, so let
> me break it down:
>
> ## foreman-users
>
> When it comes to supporting our users, what matters is that they can
> ask a question, get a reply, and feel confident in that reply. For
> those who do the replying, they need to be highlight and in some way
> rewarded for the work they do.
>
> The problem with a mailing list is that neither of these things is
> really achievable. If a user (new to our list, who knows no-one) gets
> 2 different replies, who is (s)he to trust? A forum can display user
> levels, and badges, making the developer reply stand out from the
> other new user's reply. A mailing list has nothing - and worse, the
> Groups API is so bad that I barely know who our mailing list regulars
> are (I have to webscrape it using 

Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

Greg, I have not interest in moving to any kind of web UI. I want to
send another millions of emails, including those "+1". I think we all
agree that any kind of migration must not disrupt way we work today -
a plain mailing-list we all know and use for decades.

Thanks for decreasing it. I still have concerns about the non-standard
special per-thread address way of replies which I get.

How do I start a new thread via email? Can I do that, can't I?

LZ



On Fri, Nov 3, 2017 at 10:58 AM, Greg Sutcliffe <g...@emeraldreverie.org> wrote:
> On 02/11/17 18:35, Lukas Zapletal wrote:
>> Tried to reply with just few words and I am getting:
>>
>> We’re sorry, but your email message to
>> [“theforeman.discourse+680bec16c469f36694d1ecef341e8...@gmail.com”]
>> (titled Re: [TheForeman] [Testing Area] October newsletter) didn’t
>> work.
>> Reason:
>> Body is too short (minimum is 20 characters)
>> If you can correct the problem, please try again.
>>
>> I am gonna definitely hate this, can we turn this off? A reply of
>> "yes" is still a valid one.
>
> True enough. Of course, that can be altered (seriously, there's a
> setting for *everything*) but I'll give you the Discourse reason for
> that being 20 by default:
>
> Since Discourse is intended as web-first, the idea is that instead of
> "me too", "I like this", "+1" style replies, you should be encouraged to
> click the "Like" button instead. From a discussion viewpoint, a simple
> "yes", while valid, doesn't add to the *debate*, only to the weight of a
> previous opinion, and instead of the topic author having to *manually*
> count up the '+1' posts, (s)he can just look at what got the most likes.
>
> Of course, it's tricky to click a Like by email, so I've reduced it 1
> character minimum for now. Please do test it again.
>
> Longer term, I hope we do move to using the UI features such as likes,
> polls, etc, but I accept that isn't going to happen overnight :D
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman instrumenting analysis

[Lukas Zapletal]

> I lean towards the push model here. The main reason is
> the simpler way to publish the instrumentation data from whatever
> process we want to track. Also, my understanding is, that we don't care
> only if the service is up or down (readiness and liveness) but also
> about trends during the processing.

My ultimate goal with this thread is to find an agreement and then
file a PR so Foreman Rails can send telemetry data in a way that
anyone can integrate this into any kind of existing or future
monitoring framework.

I am currently NOT interested in any kind of system monitoring,
services or performance at this point, although I already have some
plans in this regard. Once we have Rails telemetry integration, I will
likely show demo of how this works with PCP monitoring framework.

--
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Propsing a move from Google Groups to Discourse

[Lukas Zapletal]

Tried to reply with just few words and I am getting:

We’re sorry, but your email message to
[“theforeman.discourse+680bec16c469f36694d1ecef341e8...@gmail.com”]
(titled Re: [TheForeman] [Testing Area] October newsletter) didn’t
work.
Reason:
Body is too short (minimum is 20 characters)
If you can correct the problem, please try again.

I am gonna definitely hate this, can we turn this off? A reply of
"yes" is still a valid one.

LZ

On Wed, Nov 1, 2017 at 4:54 PM, Greg Sutcliffe  wrote:
> Hi all,
>
> As ever from me, this is long. Sorry about that, it's a habit. Here's
> the TL;DR:
>
> * What: move Google Groups -> Discourse
> * Why: https://blog.discourse.org/category/use-cases/
> * Can I try? - Scroll to the end for login details
>
> So, as some of you know, I'm a big fan of the Discourse[1] forum
> software - I use it for another community, and it's just lovely. I've
> been testing it out recently with a view to using it for Foreman, and
> I think it's time to explain my reasoning and ask for your thoughts.
>
> # What?
>
> Firstly, the "what" - what do I want to do? Simply put, I want to
> migrate from Google Groups to Discourse. That means locking the groups
> from further emails and using Discourse for our "written" discussions.
> Obviosuly there's data migration that needs to happen there, but we'll
> get to that.
>
> Before all the die-hard mailing list fans stop reading here - please
> keep reading. Discourse has options to interact entirely by email.
> Your workflow may not be broken :)
>
> # Why?
>
> Why do I want this? The short version is "because anything is better
> than Google Groups", but more seriously, I think Discourse is great.
> The reasons are different for each of our mailing lists though, so let
> me break it down:
>
> ## foreman-users
>
> When it comes to supporting our users, what matters is that they can
> ask a question, get a reply, and feel confident in that reply. For
> those who do the replying, they need to be highlight and in some way
> rewarded for the work they do.
>
> The problem with a mailing list is that neither of these things is
> really achievable. If a user (new to our list, who knows no-one) gets
> 2 different replies, who is (s)he to trust? A forum can display user
> levels, and badges, making the developer reply stand out from the
> other new user's reply. A mailing list has nothing - and worse, the
> Groups API is so bad that I barely know who our mailing list regulars
> are (I have to webscrape it using a crawler ...) so I can reach out to
> them for help, questions etc.
>
> Discourse also searches for likely similar topics *while* you're
> typing yours. This should hopefully help users to find existing
> support for their issue before they make duplicate posts :)
>
> For hard-working supporters, Discourse provides automatic "trust
> levels" (in additional to manually promoting admins and moderators).
> These trust levels allow you to unlock extra powers as you participate
> in the community, which helps to reward the people who do help out on
> the boards. Gamifying? yes, but it works. It'll also help me know who
> to speak to when there's some swag to send out ...
>
> To make this worse (for Groups) there is a confounding of data in the
> users-list today - we also use it for plugin announcements, and events
> & CFPs, and so on. This is because making a new mailing list would
> have a much smaller membership, making it not suitable for purpose.
> Yes, you can say "hey we created a new mailing list for X, sign up!"
> but the reality is that people just don't. Discourse gives us the
> flexibility to create new boards for things as-and-when we feel the
> need to.
>
> Discourse also supports plugins (its a rails app), so we can look at
> things like templates for new support issues, and so on. It also has
> bots which can be used to deal with some things (I have not
> investigated the bots too much yet).
>
> ## foreman-dev
>
> In contrast to the -users list, which is primarily support and
> notifications, dev is all about discussion. Here I think we'll see a
> real benefit in a couple of areas.
>
> Firstly, back when we discussed how to handle RFCs, one of the
> criticisms of a mailing list was the lack of rich format support
> (markdown, images, code snippets etc) - Discourse supports all of
> that. Quality of discussions should improve, I hope.
>
> Discourse also has "like" buttons which can be used in place of "+1"
> emails, and hopefully would encourage some of the quiter voices in the
> community to click the like button, even when they might not have
> emailed a +1. There are also polling options, so taken together I
> *hope* this will help us to resolve discussions better than we have
> historically.
>
> The ability to create dedicated boards for popular plugins is also
> pretty nice, if we decide to go that way.
>
> ## foreman-announce
>
> This list has historically been very low traffic. On Discourse, the
> most useful 

Re: [foreman-dev] Building a Rails 5.1 SCL

[Lukas Zapletal]

Big +1 to all of that.

I think COPR provides some own GPG keys and IIRC you can't override
those. It is possible to download packages from COPR and sign them
again with a custom key of course. That's perhaps your plan I guess.
Custom signatures is on COPR development TODO I think.

LZ

On Wed, Nov 1, 2017 at 7:00 PM, Eric D Helms  wrote:
> In a previous thread [1], we discussed building an SCL vs. vendorizing gems
> and the general consensus was to build an SCL. This thread is to outline a
> starting plan for how to build and maintain a Rails 5.1 SCL. I invite and
> appreciate comments towards this design.
>
> I'll begin by laying out some overall goals for the effort, a general
> proposal of information and finally a breakdown of the why for each of the
> proposal items to better explain my thinking.
>
>
> Goals
>
>  * Stand alone Rails 5.1 SCL and repository
>  * Owned and maintained by the Foreman community
>  * Easy to update and maintain
>  * Strive for automation and package tooling to reduce maintenance cost
>
>
> Proposal
>
> Build location: Copr
> SCL Name: tfm-ror51
> Git repository: theforeman/tfm-ror51-packaging
> Hosted: yum.theforeman.org/rails/tfm-ror51
> RPM Signing: signed by unique Foreman owned GPG key
> Tooling Repo: create package tooling repository separate from
> tfm-ror51-packaging repo
> Tooling Technology: Ansible
>
>
> Breakdown
>
> Build Location
>
> There has been discussion around moving our RPM builds to Copr and off of
> Koji. This will require shifting our configuration and setup, testing and
> working out kinks in Copr workflow. Building this Rails SCL provides us an
> opportunity to use Copr from the start, get familiar with it and workout
> tooling to interact with and manage a repository there. Therefore, I am
> proposing to start with Copr from the get go and avoid Koji.
>
> SCL Name
>
> Given the Foreman community will own the SCL, and our SCL namespace is 'tfm'
> I am suggesting we follow convention by naming it tfm-ror51. Previous Rails
> SCLs, were named similarly: rh-ror41, rh-ror42, sclo-ror42.
>
> Git Repository
>
> I am proposing we don't put this into foreman-packaging given the lifecycle
> of the SCL will not follow that of Foreman. Further, with the goal to create
> a stand-alone Rails SCL repository this should have its own lifecycle.
>
> Hosted
>
> We could point at the direct Copr repository, and reduce our bandwidth.
> However, since we own this Rails SCL I believe we should host it as such
> officially. Further, this would allow us to do some pre-flight testing by
> building a repository in Copr, running a test of either the SCL itself
> and/or Foreman against the SCL updates before promoting.
>
> RPM Signing
>
> I am recommending here that we sign the SCL RPMs with a new GPG key
> generated just for signing the SCL packages.
>
> Tooling
>
> With an eye towards foreman-packaging changes, I am proposing we create a
> separate git repository for all package tooling. The goal here would to
> build out new tooling to allow easier maintenance of the packaging and
> repository.  This will allow the tooling to evolve independently of either
> packaging repository.  Further, when applying this tooling to
> foreman-packaging later, the tooling would not have to be duplicated across
> branches.
>
> Tooling Technology
>
> I am proposing a centralization on Ansible as the tooling technology for a
> number of reasons. First, I feel that it has a low barrier to entry while
> providing a mix of high and low level interfaces. Secondly, Ansible allows
> orchestrating and building out more complex and directed tasks. Third, a
> team of us has some built out Ansible tooling that has been working well for
> us in another area that would be easy to port for packaging management. I
> personally think bash is complex to understand for complex actions and is
> too simple for building up a set of cohesive tooling.
>
> --
> Eric D. Helms
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman instrumenting analysis

[Lukas Zapletal]

> Does Prometheus only not work in a multi-process Rails web server? Does it
> work for a single process multi-threaded web server? This is an interesting
> roadblock given you'd expect this to affect lots of webserver across
> multiple languages out there.

Any Rails app that has multiple processes needs currently to figure
out how to deliver data to the HTTP endpoint. E.g. store it in a
database or something, which is not the best approach.

Absolutely, it lacks quite important feature right there. It stems
from the design which is pull-based.

> Yes, standard practice is to think about one container per pod (in a
> Kubernetes environment). However, there are patterns for things like log
> aggregation and monitoring such as doing a sidecar container that ensures
> co-location. The part I don't entirely get with sidecars is if I scale the
> pod to say 5, I get 5 web applications and 5 monitoring containers and that
> seems odd. Which I why I think the tendency is towards models where your
> single process/application is the end point for your metrics to be scrapped
> by an outside agent or services.
>
> I agree you want the collector to be separate, but if your web application
> is down what value would a monitoring endpoint being alive provide? The
> application would be down, thus no metrics to serve up. The other exporters
> such as the one exporting metrics about the underlying system would be
> responsible for giving system metrics. In the Kube world, this is handled by
> readiness and liveness probes for Kubenernetes to re-spin the container if
> it stops responding.

In container world, monitoring agents are running on hosts, not in
containers themselves. And collector agents can be 1:1 or 1:N (e.g.
for each container host). I am not sure I follow you here. Why you
don't see added value again? Monitoring agent without any apps
connected is as useful as ssh deamon waiting for connections.

Let me put it this way - push approach seems to be more appropriate
for multi process Ruby application than pull approach. That's what we
are discussing here, unless there are better protocols/agents I am not
aware of.

Honestly, pull approach via simple HTTP REST API seems cleaner but it
is just not good fit and also it creates other unnecessary
responsibility on the app itself. You are working on containerizing
Foreman, so it is also actually against this effort.

Anyway, let me throw another integration. Collectd has an agent (or
plugin) that opens a local socket which can be used to receive data
from other applications. I wrote Ruby client library the other day
(https://github.com/lzap/collectd-uxsock) but I believe this make no
difference than statsd - you still need a local process to gather the
data.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman instrumenting analysis

[Lukas Zapletal]

Statsd can be configured for remote transport, meaning that the
collecting agent (or aggregating process if you like) can run on
remote server (or container). It is recommended to run it either on
localhost or at least LAN, it is not a good idea to route the UDP
packets through complex environments tho as they can get lost. Also
creating a SPOF is not a good idea, but I've seen articles or comments
about having one central statsd collector for all hosts. Those people
had usually questions around scaleability because single point of
entry was getting overloaded.

There are some WIP patches for Prometheus as well giving a possibility
to have single HTTP REST endpoint for all subprocesses of a Rails app
server, but if you take a look into this (links are in my original
email) these are pretty hacky. One is creating a local shared memory
block for communication, the other is doing the same via serialized db
file. This is doing dozens of system calls per single measurement,
compared to just one or two for UDP datagram this is way too much
IMHO.

The question tho is if there is another protocol I am not aware of.
There are actually two which I both tested to be honest:

1) PCP trace API - http://pcp.io/man/man3/pmdatrace.3.html

PCP is a monitoring collecting daemon which is in most Linux distros
(and in RHEL as well) and it has a very simple API which uses TCP
connection for communication with trace agent (called PMDA trace). I
wrote a Ruby wrapper around this simple API
(https://github.com/lzap/ruby-pcptrace) and I have a working
prototype. Disadvantage is that in PCP world this API is seen as
legacy, might get removed in the future. Also aggregation is only done
for transaction type observation.

1) PCP MMV API - http://pcp.io/books/PCP_PG/html/id5213288nat.html

Another agent which uses memory mapped files for ultra-fast
communication. This is the fastest possible application
instrumentation I've seen, but it is a little bit of an overkill
primarily targeted to HPC environment. Also no aggregation is done and
there is no Ruby bindings at all. In both cases, a PCP daemon needs to
be running.

One question tho - isn't standard practice to have one container per
pod that will serve as monitoring endpoint? I am no expert with
Kubernetes, but I believe that's exactly what this technology is built
for - you can specify services and their dependency on each other. The
price we need to pay (an extra service) is balanced with better
reliability - I can imagine when Rails/Passenger stops responding you
won't be able to reach the monitoring endpoint as well thus we'd need
to maintain a separate web stack for that.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Foreman instrumenting analysis

[Lukas Zapletal]

Hello,

I am seeking for app instrumenting protocol for Foreman Rails
application that will fulfill the following requirements:

The protocol must work with multi-process server like Passneger.
The protocol can be easily integrated into Foreman Tasks and Smart Proxy.
The protocol or agent must support aggregation of time-based data
(quantiles, average).
The protocol must integrate with top three open-source monitoring frameworks.

Let me summarize my findings so far. I am looking for advice or
comments on this topic. I already worked on some prototypes, but
before I commit to some final solution, I want to be sure I will not
miss something I don't know about.

Before you send comments, please keep in mind I am not searching for
monitoring solution to integrate with. I want an application
instrumentation library (or protocol) to be able export measurements
(or telemetry data if you like) from Rails (like number or requests
processed, SQL queries, time spent in db or view, time spent rendering
a template or calling a backend system).


Prometheus


Flexible text-based protocol (alternatively protobuf) with HTTP
REST-like communication. It was designed to be pull-based, meaning
that an agent makes HTTP calls to web application which holds all
metrics until they are flushed. It was build for Prometheus monitoring
framework (Apache licenced) created by SoundCloud initially. Server
and most agents are written in Go, can run without external database
or export into 3rd party storage backends.


It looks great, but it has a major problem - the Ruby client library
(called client_ruby) does not support multi-process web servers at
all. There are some hacks but these are using local temp files or
shared memory with rather bad benchmark results (see the links down
below).


There is a possibility to push metrics into a separate component
called PushGateway, but this was created for things like cron jobs or
rake tasks. Doing multiple HTTP requests for each metric per single
app request will unlikely perform well. In the README authors have
note that this should be considered as "temporary solution".


Although Prometheus seems to have vibrant community, the Ruby library
development pace slowed down as SoundCloud "does not use many Ruby
apps anymore". But it is still a good option to have.


https://prometheus.io
https://prometheus.io/docs/instrumenting/pushing/
https://github.com/prometheus/client_ruby
https://github.com/prometheus/client_ruby/issues/9
https://github.com/prometheus/client_ruby/commits/multiprocess


OpenTSDB


OpenTSDB consists of a Time Series Daemon (TSD) as well as set of
command line utilities. Interaction with OpenTSDB is primarily
achieved by running one or more of the TSDs. Each TSD is independent.
There is no master, no shared state so you can run as many TSDs as
required to handle any load you throw at it. Each TSD uses the open
source database Hadoop/HBase or hosted Google Bigtable service to
store and retrieve time-series data.


It uses push mechanism via REST JSON API with alternative
"telnet-like" text endpoint. Although it does have some agents, it is
more used as a storage backend than end-to-end monitoring solution.


http://opentsdb.net/overview.html


Statsd


Main idea behind this instrumentation protocol is simple - get the
measurement out of the application as fast as possible using UDP
datagram. A collector agent usually runs locally, it does aggregation
and relays the measurements to target backend system. The vanilla
version does not support tagging, but there are extensions or mappings
possible to support that.


Almost all monitoring platforms has some kind of
agent/importer/exporter that talks via statsd. The original statsd
daemon was written in Perl years ago, then it was re-popularized by
node.js implementation, but there are many alternative agents from
which the most promising is statsite with very easy extensibility.


This protocol is my favourite because it plays well with multiprocess
Ruby servers or other Foreman components (all can just send UDP
packets to localhost) and it also takes all aggregation and storing
temporary data out of Ruby application. It also brings chances of
regressions in our codebase to bare minimum - in the worst case the
aggregating agent can fail but UDP packets will simply get lost
without interrupting the application. The best Ruby client library
seems to be statsd-instrument actively maintained by Shopify, it is
very small without any runtime dependency.


https://github.com/etsy/statsd/blob/master/docs/metric_types.md
https://github.com/Shopify/statsd-instrument
https://github.com/prometheus/statsd_exporter
https://github.com/statsite/statsite
https://codeascraft.com/2011/02/15/measure-anything-measure-everything/


New Relic, Instrumental, DataDog, Rollbar


All are paid services, some clients are open-source (Instrumental is
MIT licenced) but usually with not well documented protocol and worse
integration to different monitoring 

Re: [foreman-dev] Koji builder crash - days after

[Lukas Zapletal]

Guys,

I configured logwatch on Koji as well, cron output and logwatch is
currently being delivered to my Red Hat address. It looks like Postfix
is able to deliver to redhat.com emails just fine, but when I tested
this with gmail.com it did not work out.

Do we have some infra mailing list for this kind of mails? Anyone else
want to get these emails?

If we decide to puppetize, let's make sure this is under puppet control.

LZ

On Mon, Oct 30, 2017 at 11:33 AM, Greg Sutcliffe
<g...@emeraldreverie.org> wrote:
> Hey
>
> On Mon, 2017-10-30 at 10:22 +0100, Lukas Zapletal wrote:
>> After several hours of outage, we were able to bring it up by
>> mounting the volume in a temporary VM, editing /etc/fstab and
>> starting new instance.
>
> Thanks for the effort, especially on a Friday!
>
>> Started new wiki page where we have this information:
>>
>> http://projects.theforeman.org/projects/foreman/wiki/KojiSetup
>
> Good idea.
>
>> There were voices on the IRC to puppetize this server, I am not
>> against and feel free to add this to todo. It does not make much
>> sense IMHO to puppetize koji setup, but things like setting up ssh
>> keys or basic services can be useful.
>
> That was me, and yeah, just setting up the usual base classes will mean
> that the infra team have access (handy if you're away) and all the
> boring stuff is taken care of. One area that it might also help is
> adding Koji into our wider backup system (I don't think we can move the
> Koji files backup, but the Psql backups could be moved offsite, just in
> case)
>
> I can try to find time for this, but I'll need my key[1] adding in that
> case. No promises though - as you say, it's low priority.
>
> 1 http://emeraldreverie.org/about#sshkeys
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Koji Outage

[Lukas Zapletal]

I can confirm that rsync port was not in the SG, added and Ewoud
tested this. Thanks.

What happened is in a summary in a separate thread. Let's carry on
discussion there. But the good thing about this failure is we have
been able to recover quite fast. Next time this happens I think we are
able to spin off in dozens of minutes now. (Most of the time we were
waiting for a snapshot of the data EBS volume just to have a copy.)

LZ

On Mon, Oct 30, 2017 at 9:33 AM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> Turns out it was started in a new security group which didn't allow rsync.
> lzap fixed that now.
>
>
> On Sat, Oct 28, 2017 at 08:09:39PM +0200, Ewoud Kohl van Wijngaarden wrote:
>>
>> It looks like the rsync service isn't started causing our promotion
>> pipeline to fail. Could you have a look?
>>
>> On Thu, Oct 26, 2017 at 03:31:28PM +0200, Lukas Zapletal wrote:
>>>
>>> Here is an update. Restart did not help, we stopped the instance and I
>>> am following this guide to create new AMI and start it:
>>>
>>>
>>> http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html
>>>
>>> Image which is now pending completion is called ami-2ecd6b54 and it
>>> has both root EBS volume and data EBS volume (900 GB) which is the
>>> reason why this is so slow. Then we can start new AMI to see if it
>>> boots up. The instance type was i3.xlarge for the record, we want the
>>> same one which was the best performance/price ratio for Koji workload.
>>>
>>> After new koji boots up we want to recreate /mnt/tmp folder structure
>>> and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
>>> like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
>>> and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
>>> directories where koji was doing building locally. More CPU intensive
>>> flavours were more expensive so we had this IO intensive one instead
>>> which stills delivers 4 cores and 32 GB RAM which is good.
>>>
>>> On the main EBS volume (900 GB one) there is a backup directory and in
>>> this directory we should have a backup of the directory structure.
>>> There is a cron job that does this daily. It was not backing up
>>> temporary files, just directories. This should be enough to get koji
>>> daemons back online. There should be a daily backup of postgre
>>> database as well.
>>>
>>> The EBS volume snapshot is ongoing, it is required to do snapshot
>>> first and then you can create new AMI from it. I have some family
>>> business in an hour, so I am writing this summary so someone else form
>>> US timezone can carry on from here. Next step would be - start new
>>> instance, let it boot (there might be ext4 file system check - not
>>> sure if we use XFS or ext4 for the data volume - see the AMI console
>>> for boot) and then find the /mnt/tmp backups, restore the directory
>>> structure, restart (rather whole system than just koji) and it should
>>> show up. Last thing would be to associate the elastic IP.
>>>
>>>
>>>
>>> On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>>
>>>> Likely a hardware failure according to notification, our instance is
>>>> not responding. We are trying restart first.
>>>>
>>>> ***
>>>>
>>>> EC2 has detected degradation of the underlying hardware hosting your
>>>> Amazon EC2 instance associated with this event in the us-east-1
>>>> region. Due to this degradation, your instance could already be
>>>> unreachable. After 2017-10-30 00:00 UTC your instance, which has an
>>>> EBS volume as the root device, will be stopped.
>>>>
>>>> You can see more information on your instances that are scheduled for
>>>> retirement in the AWS Management Console
>>>> (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)
>>>>
>>>> * How does this affect you?
>>>> Your instance's root device is an EBS volume and the instance will be
>>>> stopped after the specified retirement date. You can start it again at
>>>> any time. Note that if you have EC2 instance store volumes attached to
>>>> the instance, any data on these volumes will be lost when the instance
>>>> is stopped or terminated as these volumes are physically attached to
>>>> the host computer
>>>>
>>>> * What do you need to do?

[foreman-dev] Koji builder crash - days after

[Lukas Zapletal]

Hello,

the reason why our Koji was out of service last week was a hardware
failure. The instance was respinned on a different hypervisor but due
to ephemeral storage mounted as swap and scratch disks the OS did not
come up and went into emergency mode. I was surprised frankly because
I expected the system to boot up (root volume was ok), anyway lesson
learned.

After several hours of outage, we were able to bring it up by mounting
the volume in a temporary VM, editing /etc/fstab and starting new
instance. I made some changes - cleaned up fstab and dropped
everything except the root volume. Everything else is configured in
rc.local now, so the instance should boot up on a different machine or
configuration just fine as long as the root volume is /dev/sda1.

Started new wiki page where we have this information:

http://projects.theforeman.org/projects/foreman/wiki/KojiSetup

There were voices on the IRC to puppetize this server, I am not
against and feel free to add this to todo. It does not make much sense
IMHO to puppetize koji setup, but things like setting up ssh keys or
basic services can be useful.

The wikipage now follows, I recommand to read on the wiki rather than
here, there might be updates already:

***

h1. Koji Setup

Our intance is running at AWS EC2 (us-east-1) as i3.xlarge instance (4
CPUs, 32 GB RAM, 900 GB SSD NVMe). It is running CentOS 7 from EBS
volume (8 GB). The account is managed by Bryan Kearney, access to the
instance has few people including Lukas Zapletal, Eric Helms and Mike
McCune. If you need to be there, contact them.

h2. Volumes and mounts

The instance has two EBS volumes attached:

* /dev/sda1 - root
* /dev/sdx - data volume (/mnt/koji available as /dev/xvdx1)

The instance must be running in a security group with ports 22, 80,
443, 873 (rsyncd), 44323 (read only monitoring PCP web app) allowed
(all IPv4 TCP).

Root EBS volume is mounted via UUID in fstab:


UUID=29342a0b-e20f-4676-9ecf-dfdf02ef6683 / xfs defaults 0 0


Note that other volumes are not present in fstab, this is to prevent
booting into emergency mode when the VM is respinned on a different
hypervisor with different or empty ephemeral or EBS storage
configuration. All the rest is mounted in /etc/rc.local:


swapon /dev/nvme0n1p1
mount /dev/nvme0n1p2 /mnt/tmp -o defaults,noatime,nodiratime
mount /dev/xvdx1 /mnt/koji -o defaults,noatime,nodiratime
hostnamectl set-hostname koji.katello.org
systemctl restart pmcd pmlogger pmwebd
mount | grep /mnt/koji && systemctl restart rsyncd
mount | grep /mnt/koji && systemctl start postgresql
systemctl start httpd
mount | grep /mnt/koji && mount | grep /mnt/tmp && systemctl start kojid
mount | grep /mnt/koji && mount | grep /mnt/tmp && systemctl start kojira


On our current VM flavour there is a local SSD NVMe storage
(/dev/nvme0n1) with two partitions created (50/50). The first one is
swap and the second one is mounted as /mnt/tmp where koji does all the
work. This volume needs to be fast, it grows over the time and
contains temporary files (built packages, build logs, support files).

The main data folder where PostgreSQL database and koji generated
repositories and external repositories are present is on EBS volume
mounted as /mnt/koji. Note this was created as ext4 which can
sometimes lead to mkfs, perhaps xfs would be better fit for our use
case.

Services required for koji (postgresql, httpd, kojid, kojira, rsyncd)
are only started if required volumes are mounted.

h2. Hostname

The instance has a floating IP, in /etc/hosts we have an entry for that:

34.224.159.44 koji.katello.org kojihub.katello.org koji kojihub

When the IP changes, make sure this does change as well.

When new instance is booted via AWS, it will have a random hostname
assigned. In the rc.local we set the hostname to koji.katello.org on
every boot.

h2. Backups

There is a cron job (/etc/cron.weekly/koji-backup) that performs two
actions every week:

Full PostgreSQL database dump into /mnt/koji/backups/postgres.

File system backup of /mnt/tmp (ephemeral storage) into
/mnt/koji/backups/ephemeral. This backup skips all files named RPM
(these are not needed), duplicity tool is used, no encryption is done.
The main purpose of this backup is to store required filesystem
structure so koji can be quickly brought up after crash. Since the
backup mostly contains directories and build logs, it is not big. To
restore that, use:

duplicity restore file:///mnt/koji/backups/ephemeral /mnt/tmp --force
--no-encryption

Both backups does not have any rotation and need to be deleted every
year. The full backup script looks like:


#!/bin/bash
/usr/bin/duplicity --full-if-older-than 1M --no-encryption -vWARNING
--exclude '/mnt/tmp/**/*rpm' /mnt/tmp
file:///mnt/koji/backups/ephemeral
date=`date +"%Y%m%d"`
filename="/mnt/koji/backups/postgres/koji_${date}.dump"
pg_dump -Fc -f "$filename" -U koji koji


h2. Updates

We are running C

Re: [foreman-dev] Jenkins can now send email

[Lukas Zapletal]

Ok waiting for:
https://github.com/theforeman/foreman_discovery/commit/473f28621aeba96bea53117611312a78b9159341

On Thu, Oct 26, 2017 at 5:08 PM, Greg Sutcliffe <g...@emeraldreverie.org> wrote:
> On Thu, 2017-10-26 at 15:53 +0200, Lukas Zapletal wrote:
>> Ok added me into the list in the post-build action.
>
> Cool, trigger a build and see if you get an email :P
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Heads up - hound rubocop version updated

[Lukas Zapletal]

Thanks Tomer!

LZ

On Thu, Oct 26, 2017 at 4:13 PM, John Mitsch  wrote:
> This was fixed on katello by https://github.com/Katello/katello/pull/7036
>
> If you have an open PR on katello, you may want to re-kick off Jenkins
>
> -John
>
> John Mitsch
> Red Hat Engineering
> (860)-967-7285
> irc: jomitsch
>
> On Wed, Oct 25, 2017 at 1:18 PM, Andrew Kofink  wrote:
>>
>> I created an issue in Katello [1] to track the changes there. We may
>> assign someone [who isn't there] during today's triage.
>>
>> [1] http://projects.theforeman.org/issues/21468
>>
>> On Wed, Oct 25, 2017 at 7:32 AM, Tomer Brisker 
>> wrote:
>>>
>>> Hi,
>>>
>>> Hound have updated their version of rubocop to 0.51 which includes some
>>> rule name changes. I have opened a PR [1] to foreman-core to upgrade to the
>>> latest version and updated the todo file.
>>> All plugins using rubocop should make sure their .rubocop.yaml is
>>> compatible with the new version.
>>>
>>> Hound ignores the rubocop version specified in the gemfile, which means
>>> that it will start barking on every PR in repos that use it if the settings
>>> are incompatible, and once the PR is merged into foreman core, those plugins
>>> depending on core and running rubocop on jenkins will start failing.
>>>
>>> The changes to rubocop's rule names can be foun in their changelog [2],
>>> or by manually running rubocop version 0.51 with your current configuration
>>> and looking out for errors. In case the errors are in the .rubocop_todo.yaml
>>> file, it can be automatically regenerated by running `rubucop
>>> --auto-gen-config` (but keep in mind this will disable any currently failing
>>> cops).
>>>
>>>
>>> [1] https://github.com/theforeman/foreman/pull/4949
>>> [2] https://github.com/bbatsov/rubocop/blob/master/CHANGELOG.md
>>>
>>> --
>>> Have a nice day,
>>> Tomer Brisker
>>> Red Hat Engineering
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Andrew Kofink
>> akof...@redhat.com
>> IRC: akofink
>> Software Engineer
>> Red Hat Satellite
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Merge rights to Katello installer

[Lukas Zapletal]

+1 nice work thanks.

LZ

On Thu, Oct 26, 2017 at 11:39 AM, Sean O'Keeffe  wrote:
> Hi,
>
> I'd like to request write access to the installer team.
>
> I've been involved with 38 PRs across what I'd consider to be the 'main'
> repos, apparently I created 15 of them and I guess I reviewed the rest.
>
> https://github.com/Katello/puppet-katello/pulls?q=is%3Apr%20involves%3Asean797%20
> https://github.com/Katello/katello-installer/pulls?utf8=%E2%9C%93=is%3Apr%20involves%3Asean797
> https://github.com/Katello/puppet-certs/pulls?utf8=%E2%9C%93=is%3Apr%20involves%3Asean797%20
> https://github.com/Katello/puppet-candlepin/pulls?utf8=%E2%9C%93=is%3Apr%20involves%3Asean797%20
> https://github.com/Katello/puppet-foreman_proxy_content/pulls?utf8=%E2%9C%93=is%3Apr%20involves%3Asean797
> https://github.com/Katello/puppet-pulp/pulls?utf8=%E2%9C%93=is%3Apr%20involves%3Asean797%20
> https://github.com/Katello/puppet-qpid/pulls?utf8=%E2%9C%93=is%3Apr%20involves%3Asean797
>
> Thanks,
> Sean
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Jenkins can now send email

[Lukas Zapletal]

Ok added me into the list in the post-build action.

LZ

On Thu, Oct 26, 2017 at 2:58 PM, Greg Sutcliffe <g...@emeraldreverie.org> wrote:
> On Thu, 2017-10-26 at 14:50 +0200, Lukas Zapletal wrote:
>> Greg, thanks for doing this. Now, honestly I do not understand a
>> single paragraph here.
>>
>> Can you show me an example JJB patch how to enable this for a plugin?
>> Ideally discovery, with my spam eating lzap+...@redhat.com :-)
>
> I haven't looked at what's needed in JJB yet, so this is all temporary
> at the moment. I made the needed change on the discovery job[1] (that
> is, add post-build action with no triggers), so if you go there now,
> you should see a Watch Job link. Try it out, let me know if it works.
>
> [1] http://ci.theforeman.org/job/test_plugin_foreman_discovery_develop/
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Koji Outage

[Lukas Zapletal]

Here is an update. Restart did not help, we stopped the instance and I
am following this guide to create new AMI and start it:

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/creating-an-ami-ebs.html

Image which is now pending completion is called ami-2ecd6b54 and it
has both root EBS volume and data EBS volume (900 GB) which is the
reason why this is so slow. Then we can start new AMI to see if it
boots up. The instance type was i3.xlarge for the record, we want the
same one which was the best performance/price ratio for Koji workload.

After new koji boots up we want to recreate /mnt/tmp folder structure
and swap. Open /mnt/fstab to see the mountpoints, the i3.xlarge has
like 950 GB ephemeral storage, but it was unused (we had 400 GB swap
and 400 GB /mnt/tmp). In the /mnt/tmp directory there was just few
directories where koji was doing building locally. More CPU intensive
flavours were more expensive so we had this IO intensive one instead
which stills delivers 4 cores and 32 GB RAM which is good.

On the main EBS volume (900 GB one) there is a backup directory and in
this directory we should have a backup of the directory structure.
There is a cron job that does this daily. It was not backing up
temporary files, just directories. This should be enough to get koji
daemons back online. There should be a daily backup of postgre
database as well.

The EBS volume snapshot is ongoing, it is required to do snapshot
first and then you can create new AMI from it. I have some family
business in an hour, so I am writing this summary so someone else form
US timezone can carry on from here. Next step would be - start new
instance, let it boot (there might be ext4 file system check - not
sure if we use XFS or ext4 for the data volume - see the AMI console
for boot) and then find the /mnt/tmp backups, restore the directory
structure, restart (rather whole system than just koji) and it should
show up. Last thing would be to associate the elastic IP.



On Thu, Oct 26, 2017 at 2:56 PM, Lukas Zapletal <l...@redhat.com> wrote:
> Likely a hardware failure according to notification, our instance is
> not responding. We are trying restart first.
>
> ***
>
> EC2 has detected degradation of the underlying hardware hosting your
> Amazon EC2 instance associated with this event in the us-east-1
> region. Due to this degradation, your instance could already be
> unreachable. After 2017-10-30 00:00 UTC your instance, which has an
> EBS volume as the root device, will be stopped.
>
> You can see more information on your instances that are scheduled for
> retirement in the AWS Management Console
> (https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)
>
> * How does this affect you?
> Your instance's root device is an EBS volume and the instance will be
> stopped after the specified retirement date. You can start it again at
> any time. Note that if you have EC2 instance store volumes attached to
> the instance, any data on these volumes will be lost when the instance
> is stopped or terminated as these volumes are physically attached to
> the host computer
>
> * What do you need to do?
> You may still be able to access the instance. We recommend that you
> replace the instance by creating an AMI of your instance and launch a
> new instance from the AMI. For more information please see Amazon
> Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
> in the EC2 User Guide. In case of difficulties stopping your
> EBS-backed instance, please see the Instance FAQ
> (http://aws.amazon.com/instance-help/#ebs-stuck-stopping).
>
> * Why retirement?
> AWS may schedule instances for retirement in cases where there is an
> unrecoverable issue with the underlying hardware. For more information
> about scheduled retirement events please see the EC2 user guide
> (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
> To avoid single points of failure within critical applications, please
> refer to our architecture center for more information on implementing
> fault-tolerant architectures: http://aws.amazon.com/architecture
>
> LZ
>
> On Thu, Oct 26, 2017 at 1:51 PM, Eric D Helms <ericdhe...@gmail.com> wrote:
>> Our Koji is currently down from a web perspective and ssh access. Please
>> don't merge anything further to -packaging until we've resolved this. All
>> actions requiring Koji repositories for testing or actions in Koji cannot be
>> performed.
>>
>> If Bryan or Lukas (since I am not sure who has AWS access to the box) could
>> investigate for us please.
>>
>> --
>> Eric D. Helms
>> Red Hat Engineering
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscrib

Re: [foreman-dev] Koji Outage

[Lukas Zapletal]

Likely a hardware failure according to notification, our instance is
not responding. We are trying restart first.

***

EC2 has detected degradation of the underlying hardware hosting your
Amazon EC2 instance associated with this event in the us-east-1
region. Due to this degradation, your instance could already be
unreachable. After 2017-10-30 00:00 UTC your instance, which has an
EBS volume as the root device, will be stopped.

You can see more information on your instances that are scheduled for
retirement in the AWS Management Console
(https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Events)

* How does this affect you?
Your instance's root device is an EBS volume and the instance will be
stopped after the specified retirement date. You can start it again at
any time. Note that if you have EC2 instance store volumes attached to
the instance, any data on these volumes will be lost when the instance
is stopped or terminated as these volumes are physically attached to
the host computer

* What do you need to do?
You may still be able to access the instance. We recommend that you
replace the instance by creating an AMI of your instance and launch a
new instance from the AMI. For more information please see Amazon
Machine Images (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
in the EC2 User Guide. In case of difficulties stopping your
EBS-backed instance, please see the Instance FAQ
(http://aws.amazon.com/instance-help/#ebs-stuck-stopping).

* Why retirement?
AWS may schedule instances for retirement in cases where there is an
unrecoverable issue with the underlying hardware. For more information
about scheduled retirement events please see the EC2 user guide
(http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-retirement.html).
To avoid single points of failure within critical applications, please
refer to our architecture center for more information on implementing
fault-tolerant architectures: http://aws.amazon.com/architecture

LZ

On Thu, Oct 26, 2017 at 1:51 PM, Eric D Helms  wrote:
> Our Koji is currently down from a web perspective and ssh access. Please
> don't merge anything further to -packaging until we've resolved this. All
> actions requiring Koji repositories for testing or actions in Koji cannot be
> performed.
>
> If Bryan or Lukas (since I am not sure who has AWS access to the box) could
> investigate for us please.
>
> --
> Eric D. Helms
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Jenkins can now send email

[Lukas Zapletal]

Greg, thanks for doing this. Now, honestly I do not understand a
single paragraph here.

Can you show me an example JJB patch how to enable this for a plugin?
Ideally discovery, with my spam eating lzap+...@redhat.com :-)

LZ

On Thu, Oct 26, 2017 at 2:21 PM, Greg Sutcliffe  wrote:
> On Thu, 2017-10-26 at 12:58 +0100, Greg Sutcliffe wrote:
>> I have enabled it, but I can't just see where to add this to my user.
>> Perhaps someone with better eyes can spot it. If it works it will
>> remove the need to update JJB so frequently (only for the permenent,
>> important notifications).
>
> As is always the case, I found it right after hitting send ;)
>
> So, there will be some JJB config - the job does need an "Editable
> Email Notification" post-build action adding. However, unless required,
> you can delete all the triggers - simply having the action there is
> enough.
>
> Once that's done, the main page of the job has a "Watch Job" link which
> you can use to add a notify for yourself of results from that job.
>
> One thing I'm seeing is that notifications appear to be in some way
> shared across jobs - adding "On failure" to job1, and then "On
> success"£ to job2, I see "On success" on job1. If that's more than just
> a visual bug, this may not be usable - in which case we can fall back
> on JJB config anyway. Test and feedback please :)
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Merge permissions to foreman-packaging

[Lukas Zapletal]

+1

On Wed, Oct 25, 2017 at 3:25 PM, Eric D Helms  wrote:
> +1 from me as well. I'll look to add you to the team come Monday if no
> objections are raised.
>
> On Tue, Oct 24, 2017 at 5:53 AM, Ohad Levy  wrote:
>>
>>
>>
>> On Mon, Oct 23, 2017 at 7:16 PM, Ewoud Kohl van Wijngaarden
>>  wrote:
>>>
>>> Hello all,
>>>
>>> To get more involved in packaging I'd like to request merge access to
>>> foreman-packaging. I've already been monitoring PRs and submitted various
>>> patches[1][2].
>>
>>
>> +1
>>>
>>>
>>> Regards,
>>> Ewoud Kohl van Wijngaarden
>>>
>>> [1]:
>>> https://github.com/theforeman/foreman-packaging/commits/rpm/develop?author=ekohl
>>> [2]:
>>> https://github.com/theforeman/foreman-packaging/commits/deb/develop?author=ekohl
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Eric D. Helms
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] OSS Europe 2017 Prague slide deck and demo video

[Lukas Zapletal]

Hey,

I visited Prague on Monday and Tuesday to give talk about Foreman. I
gave the standard talk, updated the standard slide deck a bit and gave
it a small refresh. I pre-recorded DEMO showing PXE booting Debian 9
system with Puppet installing ntpd daemon in Foreman 1.15.

The talk was not recorded unfortunately, only the main tracks were.

https://www.slideshare.net/secret/vkNxGDAIw5gJdC
https://www.youtube.com/watch?v=XsJvKdQOHV8

Slides are available in OpenOffice.org format in our graphics assets git now:

https://github.com/theforeman/foreman-graphics/tree/master/slides/2017-oss-europe-foreman

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Safemode doesn't allow to access 'block_pass' on &:present?

[Lukas Zapletal]

Hello,

I am just preparing demo for OSS Europe and I noticed that the bug in
safemode hasn't been fixed in 1.15.6 yet. That's a blocker bug we
should consider fixing in 1.15 series.

Was there a bump of safemode gem or templates? The issue is with:

Katello Kickstart Default template:
https://gist.github.com/lzap/df6c103573dba61cc5bc22d39c6201b6

LZ

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Place to put slide decks

[Lukas Zapletal]

Hey,

I have a couple of slide decks about Foreman, mostly recycled
material. At OSS Europe in Prague I am giving "the standard" talk
about Foreman on Monday, will edit the latest version of the slides
but I was wondering - do we have a place to put those OpenOffice.org
files in?

If not, shall we create another git repository? :-)

I am putting the event into our events.yaml tomorrow.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Turned off autoupdate at transifex

[Lukas Zapletal]

I would like to know what was the reason to do that. It used to work
fine for discovery so far.

I just pushed updated POT into git, I am about to branch off 10.0
tomorrow. You told me you are going to re-enable this again? Thanks

LZ

On Thu, Oct 5, 2017 at 4:18 PM, Lukas Zapletal <l...@redhat.com> wrote:
> Bryan, I don't understand. Why this was a problem? My understanding is
> that this feature takes latest and greatest source POT file and puts
> that for translations.
>
> In my workflow, I just pull translations back into PO files when do
> doing a release. This was a good workflow I think. Or maybe I am
> missing something.
>
> LZ
>
> On Thu, Oct 5, 2017 at 2:49 PM, Bryan Kearney <bryan.kear...@gmail.com> wrote:
>> I have turned off autoupdatnig of transifex resources. This was a feature
>> for transifex to pull in the latest stings from github. This would work if
>> core and all plugin writers are syncing the strings at every release.
>>
>> If you release core or a plugin and you are not doing that as part of your
>> release process can you bug me off line? I would like to make sure that I
>> did not mess anyone up, and that strings are flowing correctly from project
>> <-> transifex <-> Red Hat.
>>
>> -- bk
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Later,
>   Lukas @lzap Zapletal



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Vendorizing or Building RPMs

[Lukas Zapletal]

I agree with Tomer here, ditto.

On the other hand, the way how rubygems are being packaged makes
package patches (RPM patch file sets) really hard (in short - it's a
hack). Since I also touch packaging work here and there, I know how
much work is to maintain full stack like Ruby on Rails. So I can
imagine having RoR vendored completely, if and only if this goes into
downstream without much changes (I see this as possible problem).
There are advantages of this approach, we could easily ship more gems,
e.g. development gems, without much effort. Every gem vendored would
need to be reviewed (changelog and license) for sure, just like
others.

So I give Node vendoring +1 and both B1 and B2 proposals also +1 +1.

LZ

On Tue, Oct 17, 2017 at 9:56 AM, Tomer Brisker  wrote:
> Thanks for bringing up the discussion!
>
> As one who has been bitten multiple times by the issue of building RPMs for
> node modules, a definite +1 (or more like +1000) from me to vendoring the
> entire node_modules directory.
> This will save a lot of time and effort on the part of maintainers that will
> no longer have to wrestle with koji for every module, as well as reduce the
> load on the builders since we won't be wasting cycles building RPMs for the
> sole purpose of consuming them during our assets compilation.
>
> However, I'm not sure if this approach should also apply to ruby gems. In
> general, the number of gems and the frequency in which they are updates are
> both lower, and unlike the node RPMs they are actually used during runtime
> and not only during build time, making sense for them to be separate RPMs
> that can be in some cases even be upgraded without upgrading the core
> packages (which again doesn't apply to node modules which will require a
> recompile anyways).
>
> On Mon, Oct 16, 2017 at 4:50 PM, Michael Moll  wrote:
>>
>> Hi,
>>
>> On Mon, Oct 16, 2017 at 02:36:27PM +0100, Greg Sutcliffe wrote:
>> > > Ruby Gems
>> > >
>> > > Option 2, to vendorize, is a deviation from our prior practices in
>> > > the area of production deployment. Thus, I am reaching out to the
>> > > community to get feedback. One interesting consideration for
>> > > vendorizing is when containers are considered and having the ability
>> > > to build them using 'bundle install' versus using RPM based
>> > > installation.
>> >
>> > Vendoring hasn't (to my knowledge) caused many issues for Debian users
>> > (Michael?), and having consistent build processes makes it easier for
>> > anyone to support users on different OSs.
>>
>> From a user's perspective it's not that bad, but IMHO the current
>> approach does have some downsides regarding plugins and I'm unsure if a
>> big plugin like Katello can be handled by the current DEB way of doing
>> things.
>>
>> Regards
>> --
>> Michael Moll
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Have a nice day,
> Tomer Brisker
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Current State of Nightlies

[Lukas Zapletal]

EPEL is not great place to be for Rails or Node components. You should
not bump versions in EPEL7 (major relase should go into EPEL8).

On Tue, Oct 17, 2017 at 12:01 AM, Eric D Helms  wrote:
>
>
> On Oct 16, 2017 5:17 PM, "Sean O'Keeffe"  wrote:
>
> Why dont we ask the maintainer to pkg a new version or someone offer to
> become a co-maintainer and get a new version into EPEL ?
>
>
> While I think this is the right open source path, I'd weigh:
>
>   1) how long will nighties be broken waiting on a new package?
>   2) 2.0 to 4.1 is a large jump and as a base dependency other EPEL packages
> may not work.
>
>
>
> On Mon, Oct 16, 2017 at 9:31 PM, Ewoud Kohl van Wijngaarden
>  wrote:
>>
>> On Mon, Oct 16, 2017 at 04:18:53PM -0400, Eric D Helms wrote:
>>>
>>> Nightly RPM and tests have been broken for around 2 weeks now. This
>>> morning
>>> a bit of a regression was merged to foreman core to fix the breaking RPM
>>> aspect and I can report that nightly RPMs are now building. However, this
>>> leads to a breakage in plugin asset usage with the newer React
>>> components.
>>> To potentially address this I have opened [1] for testing and input. As
>>> part of the original breakage, I added to test_develop running npm
>>> install
>>> and webpack compile the same way our RPMs do in order to catch these sort
>>> of issues earlier.
>>
>>
>> Thanks for this!
>>
>>> The second half is that after RPMs were built, repoclosure on the test
>>> pipeline is currently failing [2]. The highlight being:
>>>
>>> *20:10:35* package: nodejs-react-dom-15.6.2-1.el7.noarch from
>>> undertest-yum_el7-4203183943-68*20:10:35*   unresolved deps:
>>> *20:10:35*  npm(object-assign) >= 0:4.1.0*20:10:35*
>>> npm(loose-envify) < 0:2*20:10:35*  npm(loose-envify) >= 0:1.1.0
>>>
>>>
>>>
>>> nodejs-object-assign-2.0.0 exists in EPEL leading me to believe we will
>>> need to add both of these as top level packages carried in
>>> foreman-packaging. Can anyone confirm or deny that is how this should be
>>> working? Or should another package we build be providing these?
>>
>>
>> I do believe we need to package this ourselves since we need a newer
>> version than in EPEL.
>>
>>> [1] https://github.com/theforeman/foreman/pull/4924
>>> [2] http://ci.theforeman.org/job/packaging_repoclosure/37110/console
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Redmine running slowly

[Lukas Zapletal]

Thanks guys for doing this.

If things go terribly wrong, we still have an account on EC2 where we
run our koji.

On Tue, Oct 10, 2017 at 9:21 PM, Greg Sutcliffe
 wrote:
> On Tue, 2017-10-10 at 10:50 -0400, Andrew Kofink wrote:
>> Me as well. It's quite difficult to work this way.
>
> Yeah, I know :(
>
> Openshift aren't saying much other than that this is mainly due to the
> number of people that decided to upgrade to Silver Tier to avoid the
> sunset of v2. That's putting a lot of load on the v2 cluster, which
> obviously is hitting us.
>
> As Ewoud said, we've made a change today in how we process the
> underlying cron jobs that should reduce the amount of IO we were doing
> - if there's any kind of quota-ing going on, that should help. We're
> seeing that bring the time taken to run the cron down to about 10mins
> (starting at the top of the hour). That should improve things during
> that period. Sadly I did make a mistake during a manual part of the
> changes that impacted the DB, but that should be resolved now.
>
> Base load now seems to be down to around 7-9 which is better but still
> too high. Sadly the v3 resources are unlikely to be available before
> November, which is a limiter. If things are not better in the next day
> or two, then on Thu or Fri I may migrate it to our Scaleway account
> anyway, as we have capacity there, although I'd rather not migrate
> twice...
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Foreman develop brakeman report

[Lukas Zapletal]

Hello,

I performed security audit report via brakeman gem and reviewed all
warnings found. None of these look like exploitable security issue to
me, so I am sending it here for further analysis.

The first two warnings really smell tho therefore I created a refactor
ticket - we should get rid of this style for the future:

http://projects.theforeman.org/issues/21267

Full report follows:

== Brakeman Report ==

Application Path: /home/lzap/work/foreman
Rails Version: 4.2.9
Brakeman Version: 4.0.1
Scan Date: 2017-10-10 13:29:23 +0200
Duration: 24.950139702 seconds
Checks Run: BasicAuth, BasicAuthTimingAttack, ContentTag, CreateWith,
CrossSiteScripting, DefaultRoutes, Deserialize, DetailedExceptions,
DigestDoS, DynamicFinders, EscapeFunction, Evaluation, Execute,
FileAccess, FileDisclosure, FilterSkipping, ForgerySetting, HeaderDoS,
I18nXSS, JRubyXML, JSONEncoding, JSONParsing, LinkTo, LinkToHref,
MailTo, MassAssignment, MimeTypeDoS, ModelAttrAccessible,
ModelAttributes, ModelSerialize, NestedAttributes,
NestedAttributesBypass, NumberToCurrency, QuoteTableName, Redirect,
RegexDoS, Render, RenderDoS, RenderInline, ResponseSplitting,
RouteDoS, SQL, SQLCVEs, SSLVerify, SafeBufferManipulation,
SanitizeMethods, SelectTag, SelectVulnerability, Send, SendFile,
SessionManipulation, SessionSettings, SimpleFormat, SingleQuotes,
SkipBeforeFilter, StripTags, SymbolDoSCVE, TranslateBug,
UnsafeReflection, ValidationRegex, WithoutProtection, XMLDoS,
YAMLParsing

== Overview ==

Controllers: 145
Models: 132
Templates: 492
Errors: 0
Security Warnings: 39

== Warning Types ==

Cross-Site Request Forgery: 2
Cross-Site Scripting: 2
Dangerous Send: 2
Dynamic Render Path: 3
File Access: 2
Mass Assignment: 1
Redirect: 1
Remote Code Execution: 4
SQL Injection: 21
SSL Verification Bypass: 1

== Warnings ==

Confidence: High
Category: Dangerous Send
Check: Send
Message: User controlled method execution
Code: host.power.send(params[:power][:action].to_sym)
File: app/controllers/hosts_controller.rb
Line: 475

Confidence: High
Category: Dangerous Send
Check: Send
Message: User controlled method execution
Code: (resource_base.friendly.find(params[:id]) or
resource_base.find_by_mac(params[:host][:mac].to_s)).power.send(params[:power_action].to_sym)
File: app/controllers/hosts_controller.rb
Line: 266

Confidence: High
Category: Remote Code Execution
Check: UnsafeReflection
Message: Unsafe reflection method constantize called with parameter value
Code: params[:host].delete(:type).constantize
File: app/controllers/hosts_controller.rb
Line: 709

Confidence: High
Category: Remote Code Execution
Check: UnsafeReflection
Message: Unsafe reflection method constantize called with parameter value
Code: params[:host].delete(:type).constantize
File: app/controllers/hosts_controller.rb
Line: 710

Confidence: High
Category: Remote Code Execution
Check: UnsafeReflection
Message: Unsafe reflection method constantize called with parameter value
Code: params[:type].constantize
File: app/controllers/api/v2/hosts_controller.rb
Line: 378

Confidence: High
Category: Remote Code Execution
Check: UnsafeReflection
Message: Unsafe reflection method constantize called with parameter value
Code: params[:type].constantize
File: app/controllers/api/v2/hosts_controller.rb
Line: 380

Confidence: High
Category: SSL Verification Bypass
Check: SSLVerify
Message: SSL certificate verification was bypassed
Code: Net::HTTP.new(URI.parse(url).host,
URI.parse(url).port).verify_mode = OpenSSL::SSL::VERIFY_NONE
File: app/models/compute_resources/foreman/model/ovirt.rb
Line: 382

Confidence: Medium
Category: Cross-Site Request Forgery
Check: ForgerySetting
Message: protect_from_forgery should be configured with 'with: :exception'
File: app/controllers/api/base_controller.rb

Confidence: Medium
Category: Cross-Site Request Forgery
Check: ForgerySetting
Message: protect_from_forgery should be configured with 'with: :exception'
File: app/controllers/application_controller.rb

Confidence: Medium
Category: File Access
Check: FileAccess
Message: Model attribute used in file name
Code: File.read(Setting[:ssl_priv_key])
File: lib/proxy_api/resource.rb
Line: 111

Confidence: Medium
Category: File Access
Check: FileAccess
Message: Model attribute used in file name
Code: File.read(Setting[:ssl_certificate])
File: lib/proxy_api/resource.rb
Line: 110

Confidence: Medium
Category: Mass Assignment
Check: MassAssignment
Message: Parameters should be whitelisted for mass assignment
Code: params[:vm].permit!
File: app/controllers/compute_resources_vms_controller.rb
Line: 39

Confidence: Medium
Category: SQL Injection
Check: SQL
Message: Possible SQL injection
Code: Host::Managed.reorder("").unscoped.authorized.group("#{resource_name}_id")
File: app/helpers/application_helper.rb
Line: 508

Confidence: Medium
Category: SQL Injection
Check: SQL
Message: Possible SQL injection
Code: User.current.widgets.where("id = #{id}")
File: app/controllers/dashboard_controller.rb
Line: 59

Confidence: Medium

Re: [foreman-dev] Katello UX nitpick - Sync Status

[Lukas Zapletal]

While I have no issues with redesigning things for better UX, I was
hoping for some suggestion (a rename perhaps) that could go into
Katello 3.4. I hit the wrong page every second day, I was wondering if
this is just me or other people also hit Sync Plans instead Sync
Status :-)

On Fri, Oct 6, 2017 at 6:53 PM, Walden Raines <wrai...@redhat.com> wrote:
>> If we had a 'repositories' page where we could just list repos across the
>> organization, that would likely be a better place to show this information
>> (and allow the user to filter to see only what are currently syncing).
>
> Sorry, I didn't articulate my idea very well.  It would basically be
> renaming the products page to "Repositories" and making that page a list of
> repositories rather than products.  Perhaps they repositories are grouped
> into products, perhaps not.  Whether or not the concept of a "product" is
> actually useful is something that interviews with users can determine.
>
>>If we were going to just yank the sync status page, i think we would need
>> to do some user interviews and confirm that the products page and dashboard
>> widgets  fulfill their needs.
>
> For sure, and I wouldn't just yank it out (as much as I'd like to) but I do
> see an opportunity to remove a page by combining its functionality into
> another.  This is a trend that I would like to continue going forward as I
> believe our current nearly 1-to-1 mapping of pages to database relations
> isn't very user friendly.  You almost have to know how the application is
> designed in order to use it effectively.  I would much rather see an "action
> based" approach to our pages where the page in the UI is a representation of
> something you want to do rather than something that exists (if that makes
> sense at all).
>
> Cheers,
> Walden
>
> On Fri, Oct 6, 2017 at 11:31 AM, Justin Sherrill <jsher...@redhat.com>
> wrote:
>>
>>
>>
>> On 10/06/2017 11:16 AM, Walden Raines wrote:
>>
>> Hey,
>>
>> As part of my campaign to reduce the number of menu items I would love to
>> remove the sync status page entirely.
>>
>> As far as I know the sync status page doesn't really provide any
>> additional functionality over the products page with the exception of being
>> able to select individual repositories from diverse products to sync (i.e.
>> the products page bulk sync forces you to sync all of the repositories in
>> each product).  I think we could add this to the products page pretty
>> easily.
>>
>> There is also a dashboard widget that shows sync status if you use the
>> sync status page just to look at the status of your syncs.
>>
>> So really what is the purpose of the sync status page?
>>
>> I believe it was so you could get a better overview of what repositories
>> were syncing.  I think the sync status dashboard widget is not currently
>> well suited for that.
>>
>> If we had a 'repositories' page where we could just list repos across the
>> organization, that would likely be a better place to show this information
>> (and allow the user to filter to see only what are currently syncing).
>>
>> If we were going to just yank the sync status page, i think we would need
>> to do some user interviews and confirm that the products page and dashboard
>> widgets  fulfill their needs.
>>
>> -Justin
>>
>>
>>
>> Cheers,
>> Walden
>>
>> On Fri, Oct 6, 2017 at 10:58 AM, Sean O'Keeffe <sokee...@redhat.com>
>> wrote:
>>>
>>> “Sync Plans” -> “Planned Sync’s” ?
>>>
>>> On Fri, 6 Oct 2017 at 15:56, Lukas Zapletal <l...@redhat.com> wrote:
>>>>
>>>> Hey,
>>>>
>>>> I constantly keep mis-clicking on Sync Plans instead Sync Status in
>>>> Katello.
>>>>
>>>> Any UX advice how to change that? I suggest either rename one to be
>>>> more visible so you can easily target that with mouse or break these
>>>> two guys into different main menu entries.
>>>>
>>>> LZ
>>>>
>>>> --
>>>> Later,
>>>>   Lukas @lzap Zapletal
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "foreman-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to foreman-dev+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You receive

[foreman-dev] Katello UX nitpick - Sync Status

[Lukas Zapletal]

Hey,

I constantly keep mis-clicking on Sync Plans instead Sync Status in Katello.

Any UX advice how to change that? I suggest either rename one to be
more visible so you can easily target that with mouse or break these
two guys into different main menu entries.

LZ

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Recording: how to setup provisioning for development

[Lukas Zapletal]

Hello,

although I know that forklift provides basic provisioning setup, this
is only for nested environment which does not work for me (e.g.
discovery livecd is booting up about an hour on my system in nested).
We had a one hour head-to-head meeting with Rahul and I created
recording of that discussing how to set things up for development of
bare-metal provisioning with libvirt, DNS, DHCP, TFTP and git
checkouts of foreman and proxy - basically without Forklilft.

https://bluejeans.com/s/acb7I

Might be useful for someone who is interested in this. Some links I
use in the video:

https://github.com/theforeman/forklift
https://github.com/theforeman/forklift/blob/master/docs/provision.md
https://lukas.zapletalovi.com/2013/04/managing-many-servers-with-foreman.html
http://downloads.theforeman.org/discovery/releases/latest/

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Redmine running slowly

[Lukas Zapletal]

Ohad mentioned this on our coffee meeting, but why we insist on
OpenShift? It's just a regular Rails app, we could easily drop it to
one of our web hosts and maybe the performance will be even better
than it was.

In other words, no OpenShift V3, our infra for now and forever. Not
sure how much work is to puppetize this, but this could be, I know
someone will eat me for these words, simply a pet server with no
automatization.

/me runs away! :-)

LZ

On Wed, Oct 4, 2017 at 6:18 PM, Greg Sutcliffe  wrote:
> Since the start of this week Redmine has been consistently slow.
> Console shows very high load (not dropping below 10, often above 20).
> We haven't chnaged anyhting, so my suspicions are on the v2 sunset that
> happened at the end of last week.
>
> I've opened a support case to see what's going on, but for now treat
> Redmine with care ;)
>
> Greg
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Re: [foreman-dev] unscheduled 1.15.5

[Lukas Zapletal]

The PR https://github.com/theforeman/foreman/pull/4555 is RTM, hope I
can make it.

Thanks

On Thu, Oct 5, 2017 at 12:57 PM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> It looks like #19986 was fixed in puppetserver 2.8.0 and GH-249 was merged
> so from the installers side I think we're good to go.
>
>
> On Wed, Oct 04, 2017 at 05:34:51PM +0200, Ewoud Kohl van Wijngaarden wrote:
>>
>> These are now released and the PR[1] is open to bump the versions. Note
>> that I want to implement a workaround for #19986[2] and backport that to
>> 1.15.5 as well.
>>
>> [1]: https://github.com/theforeman/foreman-installer/pull/249
>> [2]: http://projects.theforeman.org/issues/19986
>>
>> On Tue, Oct 03, 2017 at 12:38:55PM +0200, Ewoud Kohl van Wijngaarden
>> wrote:
>>>
>>> That's only part of it. These are still open:
>>>
>>> Fix issues on EL7.4:
>>> * https://github.com/theforeman/puppet-foreman_proxy/pull/376
>>> * https://github.com/theforeman/puppet-foreman_proxy/pull/377
>>>
>>> Fix OpenSCAP on Puppet 4 systems:
>>> * https://github.com/theforeman/puppet-puppet/pull/554
>>>
>>> Once those are merged I'll create releases and update the installer.
>>> These also fix issues downstream.
>>>
>>> On Tue, Oct 03, 2017 at 12:26:07PM +0200, Daniel Lobato Garcia wrote:
>>>>
>>>> Similarly I think the installer is meant to be updated to work with 7.4
>>>> -
>>>> https://github.com/theforeman/foreman-installer/commit/f07567ee06742155a079f076823c6a3bfbc38120
>>>>
>>>> On 10/03, Lukas Zapletal wrote:
>>>>>
>>>>> Ondra,
>>>>>
>>>>> what fixes are you talking about? Is this the container selinux
>>>>> installation issue?
>>>>>
>>>>> If there is a chance of sneaking in
>>>>>
>>>>> https://github.com/theforeman/foreman/pull/4555
>>>>>
>>>>> if this is merged in the deadline, that would be great. Pretty
>>>>> important provisioning DHCP issue there.
>>>>>
>>>>> LZ
>>>>>
>>>>> On Tue, Oct 3, 2017 at 10:35 AM, Ondrej Prazak <opra...@redhat.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> there will be a 1.15.5 Foreman release. Main motivation behind this is
>>>>>> to
>>>>>> get SELinux fixes into 1.15, but if you are aware of any critical fix
>>>>>> that
>>>>>> should go in there, let me know.
>>>>>>
>>>>>> Have a nice day,
>>>>>> Ondrej Prazak
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] unscheduled 1.15.5

[Lukas Zapletal]

Ondra,

what fixes are you talking about? Is this the container selinux
installation issue?

If there is a chance of sneaking in

https://github.com/theforeman/foreman/pull/4555

if this is merged in the deadline, that would be great. Pretty
important provisioning DHCP issue there.

LZ

On Tue, Oct 3, 2017 at 10:35 AM, Ondrej Prazak  wrote:
> Hi,
> there will be a 1.15.5 Foreman release. Main motivation behind this is to
> get SELinux fixes into 1.15, but if you are aware of any critical fix that
> should go in there, let me know.
>
> Have a nice day,
> Ondrej Prazak
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Form inputs and accessibility

[Lukas Zapletal]

I think there was a patch that added randomized IDs to all elements
which were missing it. I think it did not get it through review.

LZ

On Sat, Sep 30, 2017 at 6:33 PM, Ohad Levy  wrote:
>
>
> On Fri, Sep 29, 2017 at 7:16 PM, Walden Raines  wrote:
>>
>> Hey everyone,
>>
>> I would like to propose that we establish a requirement that all new form
>> inputs include both a unique ID attribute and a  like so:
>>
>> Name:
>> 
>>
>> For form fields where the is no label we should use bootstrap's .sr-only
>> class on the label to hide it like so:
>>
>> Name:
>> 
>>
>> This allows screen readers to relate a label field to an input in a
>> meaningful way making it easier for the visually impaired to use our
>> application (as an added bonus unique input IDs also allow for easier to
>> write and maintain QA automation).  It is important that the ID be unique to
>> the page so as not to confuse screen readers and also in order to produce
>> valid HTML.
>>
>> Please help this effort by keeping an eye out for new input fields that
>> on't have IDs and labels and if you happen to notice an input without an ID
>> and label please fix it.
>
> In foreman, its all going though a single code path, so a simple change to
>>
>>
>> https://github.com/theforeman/foreman/blob/develop/app/helpers/form_helper.rb#L326
>> will get what you are looking for.
>> See also [1].
>>
>> Thanks,
>> Walden
>>
>> [1] https://webaim.org/techniques/forms/controls#input
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Requesting discovery test for core PRs

[Lukas Zapletal]

Assuming that folks are working on CI redesign, can you guys include
that on your TODO please?

LZ

On Wed, Sep 27, 2017 at 1:46 PM, Ewoud Kohl van Wijngaarden
<ew...@kohlvanwijngaarden.nl> wrote:
> On Wed, Sep 27, 2017 at 01:34:08PM +0200, Lukas Zapletal wrote:
>>
>> what you think about adding discovery to core PRs test suite, just one
>> Ruby version on sqlite3?
>
>
> +1
>
>> Recently we had all red situation when we bumped to Rails 5. After
>> Ivan fixed that, we have the same due to fact import refactoring.
>> These changes are good, it's just bad timing when we are doing 1.16 RC
>> and these unexpected regressions happen. I would like to know in
>> advance.
>>
>> This could set a precedent, but frankly I am ok with adding more
>> plugins there. What's important I think is to run all core tests
>> without any plugin and all plugin tests (without core tests) for all
>> top-tier plugins. We don't do that, we are trying to achieve "ideal"
>> world of all combinations all rubies all db stacks individually which
>> is not realistic approach with our resources.
>
>
> In other discussions other people have suggested that only testing with
> Katello is odd and I agree. Either we test no plugins or a bunch of popular
> ones. Since testing is good, I'd prefer a bunch of popular ones.
>
> We can argue about the rules for inclusion. My suggestion is to keep it
> simple: including in the main test matrix is a commitment from both sides
> and we need at least 2 active developers willing to help with test
> failures/potential regressions in core PRs.
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Requesting discovery test for core PRs

[Lukas Zapletal]

Hello,

what you think about adding discovery to core PRs test suite, just one
Ruby version on sqlite3?

Recently we had all red situation when we bumped to Rails 5. After
Ivan fixed that, we have the same due to fact import refactoring.
These changes are good, it's just bad timing when we are doing 1.16 RC
and these unexpected regressions happen. I would like to know in
advance.

This could set a precedent, but frankly I am ok with adding more
plugins there. What's important I think is to run all core tests
without any plugin and all plugin tests (without core tests) for all
top-tier plugins. We don't do that, we are trying to achieve "ideal"
world of all combinations all rubies all db stacks individually which
is not realistic approach with our resources.

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: Database and Service Actions in RPM Post Scripts

[Lukas Zapletal]

I am all for pulling all complex changes out of post scriplets.

On the other hand, I like that not calling an installer was always an
option, at least for minor releases. We have lots of users (mainly in
downstream) who did not buy into Puppet and they tend to modify lots
of configs manually trying to avoid installer runs as possible. This
change would make this required otherwise database wont be migrated.

I would like to have an external script that would do the work but you
could still run it outside of the installer. This is kinda Satellite 5
experience, which is not bad at all I think.

LZ

On Mon, Sep 25, 2017 at 8:02 AM, Marek Hulán  wrote:
> I'd be in favor of a change and avoid running scripts in post scripts. This is
> the reason why we added "Optional Step 3 (C) - Run foreman-installer" to our
> manual [1] a long time ago. We recommend running installer after upgrade if
> users use it for initial setup. If this is too heavy step, maybe foreman-
> maintain task could be added that would ensure all is up to date.
>
> [1] https://theforeman.org/manuals/1.15/#UpgradingtoForeman1.15
>
> --
> Marek
>
> On pondělí 25. září 2017 0:54:10 CEST Andrew Schofield wrote:
>> [I'm a user, not a developer]
>>
>> I'd suggest that the RPM's *simply* drop the files onto the file system and
>> the installer then does the required actions. There are a lot of moving
>> parts to foreman and restarting one component can have impact on other
>> components. They also duplicate actions which take place in the installer.
>> The actions taken by users are (should be) yum update then a run of the
>> installer.
>>
>> On Friday, September 22, 2017 at 4:19:46 PM UTC-4, Eric Helms wrote:
>> > Howdy,
>> >
>> > There have been recent conversations that have popped up on PRs, for
>> > example [1], and IRC conversations around whether or not our RPM packages
>> > should be performing database actions and restarting services. This thread
>> > is intended to gather feedback and view points to arrive a community
>> > decision on whether or not we should continue this behavior, alter it with
>> > limitation or out right get rid of it.
>> >
>> > This mostly happens within Foreman and some plugins, and the actions
>> >
>> > performed today:
>> >  * database migrations
>> >  * database seeds
>> >  * apipie cache
>> >  * httpd restart
>> >  * foreman-tasks restart
>> >
>> > There may be others, these are the ones I am aware of. The history of
>> > these actions, as I understand it, is so that in theory you can yum
>> > install
>> > a plugin and, without further action, the Foreman server continue to run
>> > now with your plugin.
>> >
>> > Now, for my personal view point. Our application stack is fairly complex,
>> > and there are a decently large number high percentage install plugins and
>> > ecosystem of plugins in general. Plugins performing these sorta actions as
>> > part of yum install has the potential to create unintended consequences.
>> > We
>> > have created an idempotent installer to manage our server installations
>> > for
>> > a reason, to help orchestrate changes, provide a framework for known and
>> > coordinated change. And that these types of actions should be strictly
>> > relegated to it.
>> >
>> > I encourage all Foreman and plugin developers to please weigh in so that
>> > we may reach consensus.
>> >
>> > Thanks for your time and consideration.
>> >
>> >
>> > [1] https://github.com/theforeman/foreman-packaging/pull/1761
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] request for write-access to katello-packaging

[Lukas Zapletal]

+1 thanks for all the help there.

On Thu, Sep 21, 2017 at 6:47 PM, John Mitsch  wrote:
> +1 from me, you've been a great help at reviewing the katello scripts!
>
> -John
>
> John Mitsch
> Red Hat Engineering
> (860)-967-7285
> irc: jomitsch
>
> On Thu, Sep 21, 2017 at 12:00 PM, Evgeni Golov  wrote:
>>
>> Ohai,
>>
>> I would like to request write access to katello-packaging.
>>
>> I have 14 merged PRs [1] resulting in 14 commits [2].
>> If I count it right, I also somehow reviewed 19 PRs [3].
>>
>> TIA
>> Evgeni
>>
>> [1]
>> https://github.com/Katello/katello-packaging/pulls?page=1=is%3Apr+author%3Aevgeni=%E2%9C%93
>> [2] https://github.com/Katello/katello-packaging/commits?author=evgeni
>> [3]
>> https://github.com/Katello/katello-packaging/pulls?utf8=%E2%9C%93=involves%3Aevgeni%20is%3Apr%20
>>
>>
>> --
>> Beste Grüße/Kind regards,
>>
>> Evgeni Golov
>> Software Engineer
>> 
>> Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
>> Commercial register: Amtsgericht Muenchen, HRB 153243,
>> Managing Directors: Charles Cachera, Michael Cunningham, Michael
>> O'Neill, Eric Shander
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Help needed: Discovery develop broken (Rails 5)

[Lukas Zapletal]

Hello,

we are getting errors in discovery develop due to Rails 5 update:

ActiveRecord::SubclassNotFound: Invalid single-table inheritance type:
Host::Discovered is not a subclass of Host::Managed

http://ci.theforeman.org/job/test_plugin_matrix/3760/database=mysql,ruby=2.4,slave=fast/testReport/junit/(root)/DiscoveredHostsControllerTest/test_update_inheritance/

If anyone have ideas, please help me to resolve this. There are couple
of PRs pending review but I need to get tests green before. We do some
weird magic in Host and some STI changes in Rails5 regressed. I have
no idea.


-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: Bundler is unable to solve deps for fresh checkout

[Lukas Zapletal]

I just upgraded, but I think putting Gemfile.lock (even an older one
from 1.14 stable branch) could work. Weird really.

On Wed, Sep 20, 2017 at 6:57 PM, Daniel Lobato <elobat...@gmail.com> wrote:
> http://ci.theforeman.org/view/Release%20pipeline/job/release_tarballs
>
> seems to have suffered from this today (Monday 1.15.4 tarballs were built
> just fine)
>
> Did you find any other way to fix this? Switching to 2.4 could be an option,
> probably
> putting a  Gemfile.lock on every tagged release isn't a bad idea either.
>
> On Thursday, September 14, 2017 at 3:13:29 PM UTC+2, Lukas Zapletal wrote:
>>
>> Hey,
>>
>> when I do fresh foreman clone with clean Ruby version (tried 2.0.0 and
>> 2.4.1), bundle install is not able to resolve dependencies and loops
>> forever. I am using latest stable bundler, tried also pre1 version.
>>
>> Can someone confirm and provide a workaround? I think copying
>> Gemfile.lock from someone else should do it. Can someone attach a
>> pastebin me one? I currently have my workstation down (CPU in RMA) and
>> got clean setup.
>>
>> If this is confirmed, it is quite an issue for newcomers. The second
>> command new developer is asked to execute fails hard. Any suggestions?
>>
>> LZ
>>
>> --
>> Later,
>>   Lukas @lzap Zapletal
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Foreman iot platform plugin

[Lukas Zapletal]

Hello, smart-proxy is usually deployed on remote sites, datacenters or
subnets to do all "dirty work" of managing local DHCP servers, turning
servers on/off via BMC and stuff like that. Also clients connects to
smart-proxies to do their rhsm registration, yum updates or uploading
openscap reports. Actions which are usually local only, smart-proxy
uses HTTPS connection to Foreman and vice versa to communicate (proxy)
data.

It is perfect fit for your case, you want to write smart-proxy plugin
that will do the IoT work and then some plugin for code to communicate
with your smart-proxy plugin with UI if needed.

LZ

On Fri, Sep 15, 2017 at 3:17 PM, Fairouz el ouazi
 wrote:
> Hi ,
>
> This question may sound to some of you stupid , But i want to know if i
> m developing a plugin on formean to manage all the embedded devices that are
> connected to my Iot platform (LIVE Objects ) do i need a smart proxy if yes
> why and whats is the main role of smart proxy ??
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Bundler is unable to solve deps for fresh checkout

[Lukas Zapletal]

Weird, I think the problem was that I was using CenOS 7.0 or 7.1
without any updates applied.

Anyway I tried again with rbenv Ruby 2.4 and this time it worked, I
had to be setting the version incorrectly.

Thanks for help.

On Fri, Sep 15, 2017 at 12:20 PM, Ivan Necas <ine...@redhat.com> wrote:
> Tried with rhel7 VM, same gem and ruby version as yours (2.0.0p648),
> dependencies resolved just fine.
> Failed later on installation of some gem (as it requires ruby 2.1+),
> but the memory kept under 300MB
> for all the time.
>
> -- Ivan
>
> On Fri, Sep 15, 2017 at 9:08 AM, Lukas Zapletal <l...@redhat.com> wrote:
>> Currently at bundler-1.15.4. Tried both with 2.0.0 from RHEL7 and
>> latest stable Ruby.
>>
>> [root@nuc foreman]# gem --version
>> 2.0.14.1
>> [root@nuc foreman]# ruby --version
>> ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
>>
>> [root@nuc bubbles]# gem --version
>> 2.6.13
>> [root@nuc bubbles]# ruby --version
>> ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
>>
>> This is Intel NUC with 8 GB RAM, bundler runs for about a minute
>> before it takes whole memory and start swapping. Then I leave it for
>> another 5 minutes before bringing it down.
>>
>>
>> On Thu, Sep 14, 2017 at 3:36 PM, Ivan Necas <ine...@redhat.com> wrote:
>>> The easiest way to get the recent Gemfile.lock is to look at the build
>>> artifacts in jenkins jobs
>>> (also useful when investigating sudden test failures):
>>>
>>> http://ci.theforeman.org/job/test_develop/database=postgresql,ruby=2.4,slave=fast/
>>>
>>> -- Ivan
>>>
>>> On Thu, Sep 14, 2017 at 3:34 PM, Ivan Necas <ine...@redhat.com> wrote:
>>>> What errors are you seeing. I've tried with ruby 2.2.2, and 2.3.1
>>>> bundler 1.15.4, everything worked fine.
>>>> What gem --version do you have?
>>>>
>>>> Anything in `bundler.d/Gemfile.local.rb` that could be causing this?
>>>>
>>>> -- Ivan
>>>>
>>>> On Thu, Sep 14, 2017 at 3:13 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>>>> Hey,
>>>>>
>>>>> when I do fresh foreman clone with clean Ruby version (tried 2.0.0 and
>>>>> 2.4.1), bundle install is not able to resolve dependencies and loops
>>>>> forever. I am using latest stable bundler, tried also pre1 version.
>>>>>
>>>>> Can someone confirm and provide a workaround? I think copying
>>>>> Gemfile.lock from someone else should do it. Can someone attach a
>>>>> pastebin me one? I currently have my workstation down (CPU in RMA) and
>>>>> got clean setup.
>>>>>
>>>>> If this is confirmed, it is quite an issue for newcomers. The second
>>>>> command new developer is asked to execute fails hard. Any suggestions?
>>>>>
>>>>> LZ
>>>>>
>>>>> --
>>>>> Later,
>>>>>   Lukas @lzap Zapletal
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups 
>>>>> "foreman-dev" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Later,
>>   Lukas @lzap Zapletal
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to foreman-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Bundler is unable to solve deps for fresh checkout

[Lukas Zapletal]

Currently at bundler-1.15.4. Tried both with 2.0.0 from RHEL7 and
latest stable Ruby.

[root@nuc foreman]# gem --version
2.0.14.1
[root@nuc foreman]# ruby --version
ruby 2.0.0p648 (2015-12-16) [x86_64-linux]

[root@nuc bubbles]# gem --version
2.6.13
[root@nuc bubbles]# ruby --version
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]

This is Intel NUC with 8 GB RAM, bundler runs for about a minute
before it takes whole memory and start swapping. Then I leave it for
another 5 minutes before bringing it down.


On Thu, Sep 14, 2017 at 3:36 PM, Ivan Necas <ine...@redhat.com> wrote:
> The easiest way to get the recent Gemfile.lock is to look at the build
> artifacts in jenkins jobs
> (also useful when investigating sudden test failures):
>
> http://ci.theforeman.org/job/test_develop/database=postgresql,ruby=2.4,slave=fast/
>
> -- Ivan
>
> On Thu, Sep 14, 2017 at 3:34 PM, Ivan Necas <ine...@redhat.com> wrote:
>> What errors are you seeing. I've tried with ruby 2.2.2, and 2.3.1
>> bundler 1.15.4, everything worked fine.
>> What gem --version do you have?
>>
>> Anything in `bundler.d/Gemfile.local.rb` that could be causing this?
>>
>> -- Ivan
>>
>> On Thu, Sep 14, 2017 at 3:13 PM, Lukas Zapletal <l...@redhat.com> wrote:
>>> Hey,
>>>
>>> when I do fresh foreman clone with clean Ruby version (tried 2.0.0 and
>>> 2.4.1), bundle install is not able to resolve dependencies and loops
>>> forever. I am using latest stable bundler, tried also pre1 version.
>>>
>>> Can someone confirm and provide a workaround? I think copying
>>> Gemfile.lock from someone else should do it. Can someone attach a
>>> pastebin me one? I currently have my workstation down (CPU in RMA) and
>>> got clean setup.
>>>
>>> If this is confirmed, it is quite an issue for newcomers. The second
>>> command new developer is asked to execute fails hard. Any suggestions?
>>>
>>> LZ
>>>
>>> --
>>> Later,
>>>   Lukas @lzap Zapletal
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to foreman-dev+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[foreman-dev] Bundler is unable to solve deps for fresh checkout

[Lukas Zapletal]

Hey,

when I do fresh foreman clone with clean Ruby version (tried 2.0.0 and
2.4.1), bundle install is not able to resolve dependencies and loops
forever. I am using latest stable bundler, tried also pre1 version.

Can someone confirm and provide a workaround? I think copying
Gemfile.lock from someone else should do it. Can someone attach a
pastebin me one? I currently have my workstation down (CPU in RMA) and
got clean setup.

If this is confirmed, it is quite an issue for newcomers. The second
command new developer is asked to execute fails hard. Any suggestions?

LZ

-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [foreman-dev] Re: Release of Foreman Discovery 9.1.2

[Lukas Zapletal]

Hey,

foreman-discovery-image is the correct locale. We can remove the
fdi.pot from the codebase, perhaps a mis addition.

LLZ

On Fri, Sep 8, 2017 at 8:26 PM, Bryan Kearney <bryan.kear...@gmail.com> wrote:
> On 09/08/2017 08:01 AM, Lukas Zapletal wrote:
>>
>> Hello,
>>
>> new discovery is out in 1.15 repositories (Katello 3.4). We ironed out
>> few issues around kexec and migration. For more details visit:
>>
>> http://projects.theforeman.org/projects/13/issues
>>
>> Have a nice weekend.
>>
>
> QQ on this package. I see it in transifex, but there are two pot files in
> the source code:
>
> fdi.po
> foreman_discovery_image.pot
>
>
> can you please confirm that fdi.pot is not used? It appears not but i wanted
> to confirm before I tied it to downstream.
>
> -- bk
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



-- 
Later,
  Lukas @lzap Zapletal

-- 
You received this message because you are subscribed to the Google Groups 
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to foreman-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.