[logging] Re: EFK logging stack: including the Kafka output plugin and the multiline aggregation plugin

[Alessandro Menti]

Hi Rich And Louis,

> Please submit PRs against origin-aggregated-logging and
> openshift-ansible.  Note that for origin-aggregated-logging we will
> require integration testing, which may require the installation of kafka
> by the test, and some form of documentation i.e. a markdown doc for the
> docs/ subdir.

I have submitted the following PRs:

* https://github.com/openshift/openshift-ansible/pull/9668
* https://github.com/openshift/origin-aggregated-logging/pull/1302
* https://github.com/openshift/openshift-docs/pull/11642

I have marked them as [WIP] as:
- I still need to add integration testing;
- the RPMs needed by the Fluentd image still need to be added upstream.

> As far as the RPMs go, we (Red Hat devs) would like to use your spec
> files as references in order to create the packages we deliver to our
> Red Hat customers, if they are available in a public repo with an
> appropriate open source license.
SPECs and SRPMs are available at
https://github.com/AlessandroMenti/openshift-kafka-for-fluentd-rpms for
review.

Note that I have added a few more packages than those required as I
intend to submit a second set of PRs to enable Fluentd monitoring via
Prometheus, the ones required are:

- rubygem-fluent-plugin-concat
- rubygem-fluent-plugin-kafka
- rubygem-ltsv
- rubygem-poseidon_cluster
- rubygem-poseidon
- rubygem-zk
- rubygem-zookeeper

Also note that, in many cases, I omitted checks because the
prerequisites are not packaged in RHEL (I can do this, but it would
require significant additional work).

As for the license, I have obtained approval for the company I work at
to release this work under an open source license - could the MIT
license be fine (in line with the three spec files I adapted from the
Fedora ones for three packages not included in these PRs, but present in
the repo)?

Regards,
Alessandro Menti
-- 
Alessandro Menti
DevOps Expert
alessandro.me...@kiratech.it
https://www.kiratech.it/

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev

EFK logging stack: including the Kafka output plugin and the multiline aggregation plugin

[Alessandro Menti]

Hi everyone,
a customer of ours uses aggregated logging on the OpenShift platform and
has two needs:
- forwarding all logs to an external Kafka instance;
- aggregating multiline logs (e.g. Java stack traces) on the fluentd
  side before sending them to Kafka.

I have packaged the Fluentd gems providing those functionalities as RPMs
(except for a few development prerequisites required for unit testing)
and have prepared several patches to the openshift-ansible installation
playbook and to the fluentd log aggregator repository.

I would like to contribute this work upstream; since this involves
coordination between several repositories (and possibly several review
rounds), could you please give me some advice about how I should proceed?

Thanks in advance,
Alessandro Menti

___
dev mailing list
dev@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev