Support for Kubernetes metadata when using docker --log-driver=journald?

[Rich Megginson]

When using OpenShift with the standard docker log driver, there are 
container logs in the following format:


/var/log/containers/$name_of_pod-$name_of_project-$name_of_container-$containeruuid.log

The fluentd log collector uses $name_of_pod and $name_of_project to look 
up in k8s additional metadata for the pod and project (such as the uuids).


When using docker daemon --log-driver=journald, these files no longer 
exist, and the journal entries are missing $name_of_pod and 
$name_of_project in the metadata:


__CURSOR=s=7c08f275b8a545a9afb1770ef779ebb8;i=3148;b=caa653694cfa4a6ea9e4213e6cf4065b;m=13700ce9;t=53347368f6904;x=770aebf7b30ab7b3
__REALTIME_TIMESTAMP=1463755834616068
...
_COMM=docker-current
_EXE=/usr/bin/docker-current
_CMDLINE=/usr/bin/docker-current daemon 
--authorization-plugin=rhel-push-plugin --selinux-enabled 
--log-driver=journald

CONTAINER_ID=a160de6fe8fa
CONTAINER_ID_FULL=a160de6fe8facb49103c45f2bd14854b455b42153b777634ea58fb8a95124a40
CONTAINER_NAME=silly_bhaskara

When running containers via docker directly, you can use -e and -l to 
pass additional metadata through to the journal output:

  docker run -e FOOT=BALL -l LA=SF --rm --name toasted \
  --log-driver=journald --log-opt env=FOOT --log-opt labels=LA \
  fedora /bin/pwd

 MESSAGE=/
 CONTAINER_NAME=toasted
 LA=SF
 FOOT=BALL
 CONTAINER_ID=37772f57b118
 
CONTAINER_ID_FULL=37772f57b1189306fed624675cd33412461a6f04ff0f4d7a4ea5d1d41f637641

Is there some way to make k8s and/or OpenShift add fields like 
K8S_POD_NAME=$name_of_pod and

K8S_NAMESPACE_NAME=$name_of_project to the journal entries?
This will also be needed by 
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter 
which assumes the entries are tagged in a similar format to the filename 
above.  Of course if k8s/openshift can be configured to automatically 
add that additional metadata to the journal entries then the fluentd k8s 
metadata plugin might not be needed.


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Support for Kubernetes metadata when using docker --log-driver=journald?

[Rich Megginson]

On 05/20/2016 10:39 AM, Clayton Coleman wrote:

Actually, there may be an issue for this already.
https://github.com/kubernetes/kubernetes/issues/15478


I created a new issue: https://github.com/kubernetes/kubernetes/issues/25975



On Fri, May 20, 2016 at 12:38 PM, Clayton Coleman <ccole...@redhat.com> wrote:

We could support setting that on the Kubelet when we launch the
containers.  I think it's reasonable - if you open a Kube issue Derek
and I can chime in.

On Fri, May 20, 2016 at 12:12 PM, Rich Megginson <rmegg...@redhat.com> wrote:

When using OpenShift with the standard docker log driver, there are
container logs in the following format:

/var/log/containers/$name_of_pod-$name_of_project-$name_of_container-$containeruuid.log

The fluentd log collector uses $name_of_pod and $name_of_project to look up
in k8s additional metadata for the pod and project (such as the uuids).

When using docker daemon --log-driver=journald, these files no longer exist,
and the journal entries are missing $name_of_pod and $name_of_project in the
metadata:

__CURSOR=s=7c08f275b8a545a9afb1770ef779ebb8;i=3148;b=caa653694cfa4a6ea9e4213e6cf4065b;m=13700ce9;t=53347368f6904;x=770aebf7b30ab7b3
__REALTIME_TIMESTAMP=1463755834616068
...
_COMM=docker-current
_EXE=/usr/bin/docker-current
_CMDLINE=/usr/bin/docker-current daemon
--authorization-plugin=rhel-push-plugin --selinux-enabled
--log-driver=journald
CONTAINER_ID=a160de6fe8fa
CONTAINER_ID_FULL=a160de6fe8facb49103c45f2bd14854b455b42153b777634ea58fb8a95124a40
CONTAINER_NAME=silly_bhaskara

When running containers via docker directly, you can use -e and -l to pass
additional metadata through to the journal output:
   docker run -e FOOT=BALL -l LA=SF --rm --name toasted \
   --log-driver=journald --log-opt env=FOOT --log-opt labels=LA \
   fedora /bin/pwd

  MESSAGE=/
  CONTAINER_NAME=toasted
  LA=SF
  FOOT=BALL
  CONTAINER_ID=37772f57b118

CONTAINER_ID_FULL=37772f57b1189306fed624675cd33412461a6f04ff0f4d7a4ea5d1d41f637641

Is there some way to make k8s and/or OpenShift add fields like
K8S_POD_NAME=$name_of_pod and
K8S_NAMESPACE_NAME=$name_of_project to the journal entries?
This will also be needed by
https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter which
assumes the entries are tagged in a similar format to the filename above.
Of course if k8s/openshift can be configured to automatically add that
additional metadata to the journal entries then the fluentd k8s metadata
plugin might not be needed.


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: v3.6.1 and v3.7.0-rc.0 released

[Rich Megginson]

On 10/27/2017 11:43 AM, Clayton Coleman wrote:

v3.6.0 images were carried forward,


Does that mean the v3.6.0 images were tagged as-is with v3.6.1?

Can we have a "real" v3.6.1 release of logging images, meaning they are 
git tagged with v3.6.1, built, and released?



:latest was used for v3.7.0-rc.0

On Fri, Oct 27, 2017 at 7:15 PM, Rich Megginson <rmegg...@redhat.com 
<mailto:rmegg...@redhat.com>> wrote:


What about logging and metrics?

On 10/27/2017 11:09 AM, Clayton Coleman wrote:

v3.6.1 and v3.7.0-rc.0 have been released on GitHub and the
Docker Hub:

https://github.com/openshift/origin/releases/tag/v3.7.0-rc.0
<https://github.com/openshift/origin/releases/tag/v3.7.0-rc.0>
<https://github.com/openshift/origin/releases/tag/v3.7.0-rc.0
<https://github.com/openshift/origin/releases/tag/v3.7.0-rc.0>>
https://github.com/openshift/origin/releases/tag/v3.6.1
<https://github.com/openshift/origin/releases/tag/v3.6.1>
<https://github.com/openshift/origin/releases/tag/v3.6.1
<https://github.com/openshift/origin/releases/tag/v3.6.1>>

Thanks to everyone for their hard work so far!


___
dev mailing list
dev@lists.openshift.redhat.com
<mailto:dev@lists.openshift.redhat.com>
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev
<http://lists.openshift.redhat.com/openshiftmm/listinfo/dev>






___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [logging] Re: EFK logging stack: including the Kafka output plugin and the multiline aggregation plugin

[Rich Megginson]

Thanks for this!  Will follow up in the PRs.

On 08/20/2018 01:47 AM, Alessandro Menti wrote:

Hi Rich And Louis,


Please submit PRs against origin-aggregated-logging and
openshift-ansible.  Note that for origin-aggregated-logging we will
require integration testing, which may require the installation of kafka
by the test, and some form of documentation i.e. a markdown doc for the
docs/ subdir.

I have submitted the following PRs:

* https://github.com/openshift/openshift-ansible/pull/9668
* https://github.com/openshift/origin-aggregated-logging/pull/1302
* https://github.com/openshift/openshift-docs/pull/11642

I have marked them as [WIP] as:
- I still need to add integration testing;
- the RPMs needed by the Fluentd image still need to be added upstream.


As far as the RPMs go, we (Red Hat devs) would like to use your spec
files as references in order to create the packages we deliver to our
Red Hat customers, if they are available in a public repo with an
appropriate open source license.

SPECs and SRPMs are available at
https://github.com/AlessandroMenti/openshift-kafka-for-fluentd-rpms for
review.

Note that I have added a few more packages than those required as I
intend to submit a second set of PRs to enable Fluentd monitoring via
Prometheus, the ones required are:

- rubygem-fluent-plugin-concat
- rubygem-fluent-plugin-kafka
- rubygem-ltsv
- rubygem-poseidon_cluster
- rubygem-poseidon
- rubygem-zk
- rubygem-zookeeper

Also note that, in many cases, I omitted checks because the
prerequisites are not packaged in RHEL (I can do this, but it would
require significant additional work).

As for the license, I have obtained approval for the company I work at
to release this work under an open source license - could the MIT
license be fine (in line with the three spec files I adapted from the
Fedora ones for three packages not included in these PRs, but present in
the repo)?

Regards,
Alessandro Menti



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: EFK logging stack: including the Kafka output plugin and the multiline aggregation plugin

[Rich Megginson]

Please submit PRs against origin-aggregated-logging and 
openshift-ansible.  Note that for origin-aggregated-logging we will 
require integration testing, which may require the installation of kafka 
by the test, and some form of documentation i.e. a markdown doc for the 
docs/ subdir.


To communicate about this issue on this mailing list, please prefix the 
Subject: with [logging] to make sure it gets the attention of the 
logging developers.


As far as the RPMs go, we (Red Hat devs) would like to use your spec 
files as references in order to create the packages we deliver to our 
Red Hat customers, if they are available in a public repo with an 
appropriate open source license.


On 07/18/2018 05:50 AM, Louis Santillan wrote:
You can find the EFK project here 
(https://github.com/openshift/origin-aggregated-logging).


___

LOUIS P.SANTILLAN

Architect, OPENSHIFT, MIDDLEWARE & DEVOPS

Red Hat Consulting,  Container and PaaS Practice

lsant...@redhat.com   M: 3236334854

  
TRIED. TESTED. TRUSTED. 





On Wed, Jul 18, 2018 at 2:25 AM Alessandro Menti 
mailto:alessandro.me...@kiratech.it>> 
wrote:


Hi everyone,
a customer of ours uses aggregated logging on the OpenShift
platform and
has two needs:
- forwarding all logs to an external Kafka instance;
- aggregating multiline logs (e.g. Java stack traces) on the fluentd
  side before sending them to Kafka.

I have packaged the Fluentd gems providing those functionalities
as RPMs
(except for a few development prerequisites required for unit testing)
and have prepared several patches to the openshift-ansible
installation
playbook and to the fluentd log aggregator repository.

I would like to contribute this work upstream; since this involves
coordination between several repositories (and possibly several review
rounds), could you please give me some advice about how I should
proceed?

Thanks in advance,
Alessandro Menti

___
dev mailing list
dev@lists.openshift.redhat.com 
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Fwd: Re: [CentOS-devel] ETA when PaaS SIG 3.9 will fix the "SerialFileGenerator starts at 2" issue?

[Rich Megginson]

When is the next 3.9 release scheduled?  If 3.9 GA is going to be a 
while, can we have a v3.9.0-alpha.5 asap? v3.9.0-alpha.4 is not really 
usable.


origin-aggregated-logging CI for branch/backport releases relies heavily 
on centos paas sig packages - we cannot do any ci for 3.9 until this 
issue is resolved.


 Forwarded Message 
Subject: 	Re: [CentOS-devel] ETA when PaaS SIG 3.9 will fix the 
"SerialFileGenerator starts at 2" issue?

Date:   Thu, 8 Mar 2018 05:44:40 -0800
From:   Troy Dawson <tdaw...@redhat.com>
Reply-To:   The CentOS developers mailing list. <centos-de...@centos.org>
To: The CentOS developers mailing list. <centos-de...@centos.org>



Except for the master branch (currently at origin-3.10), our rpm
building automation isn't designed for individual commits.  We are
setup to only pull on git tags, or the latest master commit.

Please talk to upstream about doing an alpha5, or rc0, or whatever
they feel is next.

Troy

On Wed, Mar 7, 2018 at 2:33 PM, Rich Megginson <rmegg...@redhat.com> wrote:

The current 3.9 centos paas repos are at v3.9.0-alpha.4.  However, I cannot
deploy origin 3.9.0-alpha.4 using ansible (openshift-ansible latest
release-3.9 branch) because I run into this issue:

TASK [openshift_master : Wait for API to become available]
*
"* NSS error -8054 (SEC_ERROR_REUSED_ISSUER_AND_SERIAL)",
"* You are attempting to import a cert with the same issuer/serial
as an existing cert, but that is not the same cert.",

This was fixed by 95e262011438642410959914c3db9868c57739ff

But that's at commit v3.9.0-alpha.4-426-g95e2620114

I realize the PaaS SIG is trying to stay in sync with official origin
tags/releases, but I think that origin 3.9.0-alpha.4 simply isn't deployable
using ansible.

Alternately, is there a workaround?

___
CentOS-devel mailing list
centos-de...@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel

___
CentOS-devel mailing list
centos-de...@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [CentOS-devel] CentOS Origin packages in testing for 3.7, 3.8 and 3.9

[Rich Megginson]

On 04/13/2018 09:58 AM, Troy Dawson wrote:

On Fri, Apr 13, 2018 at 8:42 AM, Rich Megginson <rmegg...@redhat.com> wrote:

On 04/13/2018 09:39 AM, Troy Dawson wrote:

On Fri, Apr 13, 2018 at 7:58 AM, Rich Megginson <rmegg...@redhat.com>
wrote:

On 04/12/2018 07:32 AM, Troy Dawson wrote:

We have origin packages for 3.7.2, 3.8.0 and 3.9.0.  We also have the
corresponding openshift-ansible packages. They have been put in our
testing repos.

DO NOT USE ORIGIN 3.8.0, IT IS FOR UPGRADE PURPOSES ONLY

These will not be released until *someone* has tested them.  So
please, someone, anyone, test them, and let us know.

origin 3.9 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin39/
origin-3.9.0-1.el7.git.0.ba7faec
openshift-ansible-3.9.0-0.53.0.git.1.af49d87.el7

origin 3.8 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin38/
origin-3.8.0-1.el7.git.0.dd1558c
openshift-ansible-3.8.37-1.git.1.151d57f.el7

origin 3.7 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin37/
origin-3.7.2-1.el7.git.0.cd74924
openshift-ansible-3.7.43-1.git.1.ed51ddd.el7

Once again, test and let us know.  If you don't know who to send it
to, just reply to this email.


Having some problems with logging Kibana certificates - issued with
duplicate serial numbers - was

https://github.com/openshift/origin/pull/18713/commits/95e262011438642410959914c3db9868c57739ff
fixed in these 3.9 packages?

  From what I can tell, yes.
Is it acting like it isn't?


It is the same symptom - certificates are being issued with duplicate serial
numbers - but if the fix is in origin 3.9, I'll see if there could possibly
be another cause.


I just double checked the code itself, just in case some later commit
changed things.
The changes that are in
https://github.com/openshift/origin/pull/18713/commits/95e262011438642410959914c3db9868c57739ff
are in origin-3.9.0-1.el7.git.0.ba7faec
So ya, there must be some other cause.


Seems to be working now.  Not sure what happened - probably didn't get 
the latest versions of all of the packages last time I tried.


Sorry for the noise.



Troy



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [CentOS-devel] CentOS Origin packages in testing for 3.7, 3.8 and 3.9

[Rich Megginson]

On 04/13/2018 09:39 AM, Troy Dawson wrote:

On Fri, Apr 13, 2018 at 7:58 AM, Rich Megginson <rmegg...@redhat.com> wrote:

On 04/12/2018 07:32 AM, Troy Dawson wrote:

We have origin packages for 3.7.2, 3.8.0 and 3.9.0.  We also have the
corresponding openshift-ansible packages. They have been put in our
testing repos.

DO NOT USE ORIGIN 3.8.0, IT IS FOR UPGRADE PURPOSES ONLY

These will not be released until *someone* has tested them.  So
please, someone, anyone, test them, and let us know.

origin 3.9 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin39/
origin-3.9.0-1.el7.git.0.ba7faec
openshift-ansible-3.9.0-0.53.0.git.1.af49d87.el7

origin 3.8 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin38/
origin-3.8.0-1.el7.git.0.dd1558c
openshift-ansible-3.8.37-1.git.1.151d57f.el7

origin 3.7 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin37/
origin-3.7.2-1.el7.git.0.cd74924
openshift-ansible-3.7.43-1.git.1.ed51ddd.el7

Once again, test and let us know.  If you don't know who to send it
to, just reply to this email.


Having some problems with logging Kibana certificates - issued with
duplicate serial numbers - was
https://github.com/openshift/origin/pull/18713/commits/95e262011438642410959914c3db9868c57739ff
fixed in these 3.9 packages?

 From what I can tell, yes.
Is it acting like it isn't?


It is the same symptom - certificates are being issued with duplicate 
serial numbers - but if the fix is in origin 3.9, I'll see if there 
could possibly be another cause.




Troy



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [CentOS-devel] CentOS Origin packages in testing for 3.7, 3.8 and 3.9

[Rich Megginson]

On 04/12/2018 07:32 AM, Troy Dawson wrote:

We have origin packages for 3.7.2, 3.8.0 and 3.9.0.  We also have the
corresponding openshift-ansible packages. They have been put in our
testing repos.

DO NOT USE ORIGIN 3.8.0, IT IS FOR UPGRADE PURPOSES ONLY

These will not be released until *someone* has tested them.  So
please, someone, anyone, test them, and let us know.

origin 3.9 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin39/
origin-3.9.0-1.el7.git.0.ba7faec
openshift-ansible-3.9.0-0.53.0.git.1.af49d87.el7

origin 3.8 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin38/
origin-3.8.0-1.el7.git.0.dd1558c
openshift-ansible-3.8.37-1.git.1.151d57f.el7

origin 3.7 testing
https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin37/
origin-3.7.2-1.el7.git.0.cd74924
openshift-ansible-3.7.43-1.git.1.ed51ddd.el7

Once again, test and let us know.  If you don't know who to send it
to, just reply to this email.


Having some problems with logging Kibana certificates - issued with 
duplicate serial numbers - was 
https://github.com/openshift/origin/pull/18713/commits/95e262011438642410959914c3db9868c57739ff 
fixed in these 3.9 packages?

Thanks
Paas Sig Group
___
CentOS-devel mailing list
centos-de...@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: [CentOS-devel] [CentOS PaaS SIG]: Origin v3.11 rpms available for testing

[Rich Megginson]

On 10/17/18 3:38 AM, Daniel Comnea wrote:

Hi,

We would like to announce that OKD v3.11 rpms are available for testing at [1].

As such we are calling for help from community to start testing and let us know 
if there are issues with the rpms and its dependencies.

And in the spirit of transparency see below the plan to promote the rpms to 
mirror.centos.org repo:

 1. in the next few days the packages should be promoted to the test repo [2] 
(currently it does not exist, we are waiting to be sync'ed in the background)
 2. in one/two weeks time if we haven't heard any issues/ blockers we are going 
to promote to [3] repo (currently it doesn't exist, it will once the rpm will 
be promoted and signed)


Please note the ansbile version use (and supported) /*must be*/ 2.6.x and not 
2.7, if you opt to ignore the warning you will run into issues.

On a different note the CentOS Infra team are working hard (thanks !) to 
package and release a centos-ansible rpm which we'll promote in our PaaS repos.



So does that mean we cannot test OKD v3.11 yet, unless we build our own version 
of ansible 2.6.x?


What will happen if we attempt to use ansible 2.7?  I my testing, I get stuck 
at deploying the control plane pods - it seems the virtual networking was not 
set up by openshift-ansible.




The rational is to bring more control around the ansible version used/ required by OpenShift-ansible installer and not rely on the latest ansbile version pushed to epel repo we caused 
friction recently (reflected on our CI as well as users reporting issues)



Thank you,
PaaS SiG team

[1] https://cbs.centos.org/repos/paas7-openshift-origin311-testing/
[2] https://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin311/
[3] http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin311/

___
CentOS-devel mailing list
centos-de...@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Logging module settings when installing okd 3.11.

[Rich Megginson]

https://github.com/openshift/release/pull/2042

On 11/13/18 6:17 AM, devops036 devops036 wrote:
When installing okd 3.11 with the recently made available CentOS PaaS repo (http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin311/), I can't activate the logging module 
(openshift_logging_install_logging=true).


When investigating, the playbooks are searching for docker.io/openshift/origin-logging-elasticsearch5:v3.11 , 
docker.io/openshift/origin-logging-kibana5:v3.11  and docker.io/openshift/origin-logging-curator5:v3.11 
 container images. However, these images don't exist with the v3.11 tag.


To get the logging module installed, I have to remove the 5 at the end of each 
container image as follows:
openshift_logging_elasticsearch_image=docker.io/openshift/origin-logging-elasticsearch:v3.11
 
openshift_logging_kibana_image=docker.io/openshift/origin-logging-kibana:v3.11 

openshift_logging_curator_image=docker.io/openshift/origin-logging-curator:v3.11 


Is it normal?

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: CI automation location for RPMs is moving

[Rich Megginson]

Was this ever fixed?  Is this the cause of the website being currently 
unresponsive?


On 9/10/18 2:33 PM, Clayton Coleman wrote:

Interesting, might be an HAProxy router bug.  Can you file one?

On Mon, Sep 10, 2018 at 3:08 PM Seth Jennings mailto:sjenn...@redhat.com>> wrote:

There is a bug in the webserver configuration.  Main page links to 
https://rpms.svc.ci.openshift.org/openshift-origin-v3.11 which gets redirected 
to
http://rpms.svc.ci.openshift.org:8080/openshift-origin-v3.11/ (drops https 
and adds port number).

On Sat, Sep 8, 2018 at 9:27 PM Clayton Coleman mailto:ccole...@redhat.com>> wrote:

Previously, all RPMs used by PR and the test automation or Origin were 
located in GCS.  Starting with 3.11 and continuing forward, RPMs will be served from 
the api.ci 
cluster at:

https://rpms.svc.ci.openshift.org

You can get an rpm repo file for a release by clicking on one of the 
links on the page above or via curling the name directly:

    $ curl https://rpms.svc.ci.openshift.org/openshift-origin-v3.11.repo 
> /etc/yum.repos.d/openshift-origin-3.11.repo

The contents of this repo will be the same as the contents of the image:

docker.io/openshift/origin-artifacts:v3.11 


in the /srv/repo dir.

PR jobs for 3.11 and onwards will now use this URL to fetch content.  
The old location on GCS will no longer be updated as we are sunsetting the jobs 
that generated and used that content
___
dev mailing list
dev@lists.openshift.redhat.com 
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: CI automation location for RPMs is moving

[Rich Megginson]

On 10/9/18 8:14 PM, Clayton Coleman wrote:

What website?


curl -Lvsk https://rpms.svc.ci.openshift.org/openshift-origin-v3.11 just hangs


Just use a slash at the end - all the CI jobs look like
their working



Yep - curl -Lvsk https://rpms.svc.ci.openshift.org/openshift-origin-v3.11/ 
works fine.


Thanks!



On Oct 9, 2018, at 10:10 PM, Rich Megginson  wrote:

Was this ever fixed?  Is this the cause of the website being currently 
unresponsive?



On 9/10/18 2:33 PM, Clayton Coleman wrote:
Interesting, might be an HAProxy router bug.  Can you file one?

On Mon, Sep 10, 2018 at 3:08 PM Seth Jennings mailto:sjenn...@redhat.com>> wrote:

There is a bug in the webserver configuration.  Main page links to 
https://rpms.svc.ci.openshift.org/openshift-origin-v3.11 which gets redirected 
to
http://rpms.svc.ci.openshift.org:8080/openshift-origin-v3.11/ (drops https 
and adds port number).

On Sat, Sep 8, 2018 at 9:27 PM Clayton Coleman mailto:ccole...@redhat.com>> wrote:

Previously, all RPMs used by PR and the test automation or Origin were 
located in GCS.  Starting with 3.11 and continuing forward, RPMs will be served from 
the api.ci <http://api.ci>
cluster at:

https://rpms.svc.ci.openshift.org

You can get an rpm repo file for a release by clicking on one of the 
links on the page above or via curling the name directly:

$ curl https://rpms.svc.ci.openshift.org/openshift-origin-v3.11.repo 
> /etc/yum.repos.d/openshift-origin-3.11.repo

The contents of this repo will be the same as the contents of the image:

docker.io/openshift/origin-artifacts:v3.11 
<http://docker.io/openshift/origin-artifacts:v3.11>

in the /srv/repo dir.

PR jobs for 3.11 and onwards will now use this URL to fetch content.  
The old location on GCS will no longer be updated as we are sunsetting the jobs 
that generated and used that content
___
dev mailing list
dev@lists.openshift.redhat.com <mailto:dev@lists.openshift.redhat.com>
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Elasticsearch - 5.x

[Rich Megginson]

On 3/21/19 1:33 PM, Neale Ferguson wrote:


On 3/21/19, 15:33, "Neale Ferguson"  wrote:

 Right. The image is now 
https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there 
are similar changes for origin-logging-curator5, origin-logging-kibana5,
 origin-logging-fluentd
 
 What version of openshift-ansible did you use to deploy logging?

 3.11. Strangely, I didn't set the preview option in the hosts file but the only 
images available for me (the ones that I built) are the "5" series and these 
are the ones that were started.



Elasticsearch 5 is _not_ tech preview in 3.11 - it is fully supported - 2.x is 
gone


  
 
 



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Elasticsearch - 5.x

[Rich Megginson]

On 3/21/19 1:56 PM, Neale Ferguson wrote:

 > - So this leads to the question of how to debug the elasticsearch 
container not coming up. The console log information in the 1st email came from me 
running it manually so I could add the -v option to the elasticsearch command.  
Setting the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess 
I need to try and add the -v option to the run.sh script.
 
 This simply should not be happening.  Did you use openshift-ansible to upgrade from 3.10 to 3.11, or did you deploy logging from scratch on 3.11?

- Initially, I created the 3.11 cluster from scratch  but without logging. I 
then added the logging statements to the hosts file and then ran the logging 
playbook.
  



Are you asking how to erase everything and start over?

Can you share your inventory files with the logging parameters (be sure to 
redact any sensitive information)?




___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Elasticsearch - 5.x

[Rich Megginson]

On 3/21/19 2:31 PM, Neale Ferguson wrote:

 Are you asking how to erase everything and start over?
- No, just how to get information out of the elasticsearch container to tell me 
why it's failing.
 
 Can you share your inventory files with the logging parameters (be sure to redact any sensitive information)?


- The configuration was an all-in-one when it was first created. I added a 
compute node shortly after then added the logging.


I don't see anything obviously wrong.  I guess I'd have to view the detailed 
ansible log (e.g. ansible-playbook -vvv) to see what went wrong.



[nfs]
okcd-master.sinenomine.net

# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd
nfs
new_nodes

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 
'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider',}]
openshift_master_htpasswd_users={}
os_firewall_use_firewalld=True
openshift_logging_install_logging=true
openshift_logging_storage_kind=nfs
openshift_logging_storage_access_modes=['ReadWriteOnce']
openshift_logging_storage_nfs_directory=/exports
openshift_logging_storage_nfs_options='*(rw,root_squash)'
openshift_logging_storage_volume_name=logging
openshift_logging_storage_volume_size=200Mi
openshift_logging_storage_labels={'storage': 'logging'}
openshift_logging_kibana_hostname=logging.origin.z
openshift_logging_es_nodeselector={'node-role.kubernetes.io/infra': 'true'}
ansible_ssh_user=root
openshift_deployment_type=origin
oreg_url=docker.io/clefos/origin-${component}:${version}
openshift_examples_modify_imagestreams=true
openshift_web_console_prefix=docker.io/clefos/
openshift_disable_check=disk_availability,docker_storage,memory_availability
openshift_hosted_router_selector='node-role.kubernetes.io/infra=true'
openshift_hosted_manage_registry=true
openshift_enable_unsupported_configurations=True
openshift_hosted_registry_storage_kind=nfs
openshift_hosted_registry_storage_nfs_directory=/exports
openshift_hosted_registry_storage_nfs_options='*(rw,root_squash)'
openshift_hosted_registry_storage_volume_name=registry
openshift_hosted_registry_storage_volume_size=5Gi
openshift_cockpit_deployer_image=docker.io/clefos/cockpit-kubernetes:latest
openshift_console_install=False

# host group for masters
[masters]
okcd-master.sinenomine.net

# host group for etcd
[etcd]
master.example.com

# host group for nodes, includes region info
[nodes]
master.example.com openshift_node_group_name='node-config-all-in-one'
node.example.com openshift_node_group_name='node-config-compute'

# Adding new node to the cluster
[new_nodes]


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Elasticsearch - 5.x

[Rich Megginson]

On 3/21/19 1:41 PM, Neale Ferguson wrote:

 elasticsearch 5 is not tech preview in 3.11 - it is fully supported - and 
2.x is gone
- Understood. Did that change with the move from 3.10 to 3.11?



Yes



I must've been scanning another host which was running 3.10 to spot those 
preview vars.
- So this leads to the question of how to debug the elasticsearch container not 
coming up. The console log information in the 1st email came from me running it 
manually so I could add the -v option to the elasticsearch command.  Setting 
the DEBUG and LOGLEVEL environment variables wasn't illuminating. I guess I 
need to try and add the -v option to the run.sh script.



This simply should not be happening.  Did you use openshift-ansible to upgrade 
from 3.10 to 3.11, or did you deploy logging from scratch on 3.11?


 
 



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

Re: Elasticsearch - 5.x

[Rich Megginson]

On 3/21/19 1:23 PM, Neale Ferguson wrote:


If I want to use the techpreview elasticsearch5 images are there any other 
changes I need to make as I am seeing this on startup:

Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes

configuration like the elasticsearch.yaml, in system properties or command line

arguments.In order to upgrade all indices the settings must be updated via the

/${index}/_settings API. Unless all settings are dynamic all indices must be 
closed

in order to apply the upgradeIndices created in the future should use index 
templates

to set default values.

Please ensure all required values are updated on all indices by executing:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{

"index.number_of_replicas" : "0",

"index.number_of_shards" : "1",

"index.translog.flush_threshold_period" : "5m",

"index.translog.flush_threshold_size" : "256mb",

"index.unassigned.node_left.delayed_timeout" : "2m"

}'

*

[2019-03-21T18:22:36,493][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] 
[logging-es-data-master-1jbe2qib] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: 
java.lang.IllegalArgumentException: node settings must not contain any index 
level settings

The Dockerfile is building 5.6.10 but the 3.11.0 image from docker.io/openshift 
appears to be based on 2.4.4 (which was changed to 5.x in June 2018).



Right. The image is now https://hub.docker.com/r/openshift/origin-logging-elasticsearch5/tags - there are similar changes for origin-logging-curator5, origin-logging-kibana5, 
origin-logging-fluentd


What version of openshift-ansible did you use to deploy logging?



Neale


___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev



___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev