On 02/08/2017 01:38 PM, David Moreau Simard wrote:
Attila (cc) would know how it works with Swift but he told me this week
he had some issues with it.
Indeed the swift support seems to be quite bad. I had an issue with
1) slow upload for a lot of small files using the swift CLI client
(think 90 minutes vs. 15 seconds with rsync) and
2) os_loganalyze not being able to stream big files without errors. It
corrupted big tar files when downloading.
What we ended up doing is to use os_loganalyze as a fronted to colorize
logs but using local storage (we added a volume to the VM).
Here's my role that sets it up:
https://github.com/rdo-infra/ansible-role-logserver
Attila
David Moreau Simard
Senior Software Engineer | Openstack RDO
dmsimard = [irc, github, twitter]
On Feb 8, 2017 1:27 PM, "Javier Pena" <[email protected]
<mailto:[email protected]>> wrote:
----- Original Message -----
> Hi,
>
> Following a discussion with David about os_loganalyze [1] I wanted
to discuss
> with you about the features offered by it and if that is relevant
to include
> it and
> if yes how :)
>
> Basically this is a middleware to configure into Apache that
enhances txt or
> zipped logs
> files when requested by the user. It enhances by adding clickable
timestamp
> (to be
> easy shareable by a link) also it highlight some lines according
to the log
> severities
> and finally you can also filter by severity. So features offered
by it are
> really simple but seems convenient.
>
> As I said it works within Apache so logs should be available
locally but
> according to David someone wrote an Apache conf that make it work
> with log stored in Swift.
>
Apparently, os_loganalyze has Swift support, but I've been trying to
test it and it looks outdated. Maybe we could get it up to date.
Javier
> Upstream on the OpenStack gerrit clicking on the CI link of a
review redirect
> directly to the logs enhanced by os_loganalize [2].
>
> We already have some stuff in SF that can cover the mentioned
features:
> * The timestamp plugin of Jenkins
> * Kibana (ELK)
>
> The latter offers far more features. Sharing a given log is easily
doable
> as well as filtering but that less user friendly, the kibana
interface is not
> so easy to deal with (there is a learning curve) :) Here is an
example of
> query [3]
>
> The former, this an example here [4], adds a timestamp in front of
logs but
> I guess that all you can't expect from it.
>
> So IMO if we want to have it in SF logs needs to be exported by
default as
> files in a standard FS of SF and enable the middleware in the
Apache gateway.
> As swift log export is optional I don't see an obvious solution to
manage it
> out of the box (maybe forcing swift container access via the gateway).
>
> But at the end I think we already have an out of the box solution
to store
> logs and even better a solution to search through logs :) ELK. Why not
> activating
> that by default and enhance hideci.js (on gerrit) to also give
links to
> kibana
> interface with the right query for a given job ?
>
> Do you have any suggestions ?
>
> Cheers,
> Fabien
>
> 1: https://github.com/openstack-infra/os-loganalyze
<https://github.com/openstack-infra/os-loganalyze>
> 2: https://review.openstack.org/#/c/424947/
<https://review.openstack.org/#/c/424947/>
> 3:
>
https://softwarefactory-project.io/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:%272017-02-08T11:41:30.000Z%27,mode:absolute,to:%272017-02-08T11:42:00.000Z%27))&_a=(columns:!(message),filters:!((%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%27logstash-*%27,key:%27@timestamp%27,negate:!f,value:%27February%208th%202017,%2006:41:46.630%27),query:(match:(%27@timestamp%27:(query:1486554106630,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%27logstash-*%27,key:message,negate:!f,value:%27%20%20%20%20sf-resources:%20allow-delete%27),query:(match:(message:(query:%27%20%20%20%20sf-resources:%20allow-delete%27,type:phrase))))),index:%27logstash-*%27,interval:auto,query:(query_string:(analyze_wildcard:!t,query:%27*%27)),sort:!(%27@timestamp%27,desc))
<https://softwarefactory-project.io/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:%272017-02-08T11:41:30.000Z%27,mode:absolute,to:%272017-02-08T11:42:00.000Z%27))&_a=(columns:!(message),filters:!((%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%27logstash-*%27,key:%27@timestamp%27,negate:!f,value:%27February%208th%202017,%2006:41:46.630%27),query:(match:(%27@timestamp%27:(query:1486554106630,type:phrase)))),(%27$state%27:(store:appState),meta:(alias:!n,disabled:!f,index:%27logstash-*%27,key:message,negate:!f,value:%27%20%20%20%20sf-resources:%20allow-delete%27),query:(match:(message:(query:%27%20%20%20%20sf-resources:%20allow-delete%27,type:phrase))))),index:%27logstash-*%27,interval:auto,query:(query_string:(analyze_wildcard:!t,query:%27*%27)),sort:!(%27@timestamp%27,desc))>
> 4:
>
https://softwarefactory-project.io/jenkins/job/software-factory-unit-tests/5042/console
<https://softwarefactory-project.io/jenkins/job/software-factory-unit-tests/5042/console>
>
> _______________________________________________
> Softwarefactory-dev mailing list
> [email protected] <mailto:[email protected]>
> https://www.redhat.com/mailman/listinfo/softwarefactory-dev
<https://www.redhat.com/mailman/listinfo/softwarefactory-dev>
>
_______________________________________________
Softwarefactory-dev mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/softwarefactory-dev
